Method and apparatus for an adaptive and interactive teaching of playing a musical instrument

This application is a Continuation-in-Part of U.S. non-provisional patent application Ser. No. 17/388,050, filed 29 Jul. 2021, which is related to and claims priority from U.S. provisional patent application No. 63/120,434, filed 2 Dec. 2020, and U.S. provisional patent application No. 63/162,823, filed 18 Mar. 2021, all of which are incorporated herein by reference in their entirety.

This disclosure relates generally to an apparatus and method for teaching of playing of a musical instrument.

Unless otherwise indicated herein, the materials described in this section are for the purpose of generally presenting the background context of the disclosure and are not prior-art to the claims in this application, and are not expressly or impliedly admitted to be prior art by inclusion in this section.

FIG. 1 shows a block diagram that illustrates a system 10 including a computer system 11 , and an associated Internet 22 connection. Such configuration is typically used for computers (hosts) connected to the Internet 22 and executing a server, or a client (or a combination) software. The computer system 11 may be used as a portable electronic device such as a notebook/laptop computer, a media player (e.g., MP3 based or video player), a desktop computer, a laptop computer, a cellular phone, a smartphone, a tablet, or a Personal Digital Assistant (PDA), an image processing device (e.g., a digital camera or video recorder), any other handheld or fixed location computing devices, or a combination of any of these devices. Note that while FIG. 1 illustrates various components of the computer system 11 , it is not intended to represent any particular architecture or manner of interconnecting the components.

Network computers, handheld computers, cell phones and other data processing systems that have fewer or more components, may also be used. For example, the computer system 11 of FIG. 1 may be any personal computer. The computer system 11 may include a bus 13 , an interconnect, or other communication mechanism for communicating information, and a processor 12 , commonly in the form of an integrated circuit, coupled to the bus 13 for processing information, and for executing the computer executable instructions. The computer system 11 may also include a main memory 15 a , such as a Random Access Memory (RAM), or other dynamic storage device, coupled to the bus 13 for storing information and instructions to be executed by the processor 12 . The main memory 15 a also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 12 .

The computer system 11 further includes a Read Only Memory (ROM) 15 b (or other non-volatile memory) or other static storage device coupled to the bus 13 for storing static information and instructions for the processor 12 . A storage device 15 c , may comprise a magnetic disk or optical disk, such as a hard disk drive (HDD) for reading from and writing to a hard disk, a Solid State Drive (SSD) for reading from and writing to a solid state disk, a flash storage for reading and writing from flash drive, a magnetic disk drive for reading from and writing to a magnetic disk, an optical disk drive (such as DVD) for reading from and writing to a removable optical disk, or any combination thereof, that is coupled to the bus 13 for storing information and instructions. The hard disk drive, magnetic disk drive, and optical disk drive may be connected to the system bus 13 by a hard disk drive interface, a magnetic disk drive interface, and an optical disk drive interface, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer readable instructions, data structures, program modules and other data for the general-purpose computing devices.

Typically, the computer system 11 includes an Operating System (OS) stored in a non-volatile storage 15 b for managing the computer resources and provides the applications and programs with access to the computer resources and interfaces. An operating system commonly processes system data and user input, and responds by allocating and managing tasks and internal system resources, such as controlling and allocating memory, prioritizing system requests, controlling input and output devices, facilitating networking and managing files. Non-limiting examples of operating systems are Microsoft Windows, Mac OS X, and Linux.

The computer system 11 may be coupled via the bus 13 to a display 17 , such as a Liquid Crystal Display (LCD), a flat screen monitor, a touch screen monitor or similar means for displaying text and graphical data to a user. The display 17 may be connected via a video adapter for supporting the display. The display 17 allows a user to view, enter, and/or edit information that is relevant to the operation of the system 10 . An input device 18 , including alphanumeric and other keys, is coupled to the bus 13 for communicating information and command selections to the processor 12 . Another type of user input device is a cursor control 18 a , such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor 12 and for controlling cursor movement on the display 17 . This cursor control 18 a typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

The computer system 11 may be used for implementing the methods and techniques described herein. According to one embodiment, these methods and techniques are performed by the computer system 11 in response to the processor 12 executing one or more sequences of one or more instructions contained in the main memory 15 a . Such instructions may be read into the main memory 15 a from another computer-readable medium, such as the storage device 15 c . Execution of the sequences of instructions contained in the main memory 15 a causes the processor 12 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the arrangement. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.

The term “processor” is used herein to include, but not limited to, any integrated circuit or any other electronic device (or collection of electronic devices) capable of performing an operation on at least one instruction, including, without limitation, a microprocessor (μP), a microcontroller (μC), a Digital Signal Processor (DSP), or any combination thereof. A processor, such as the processor 12 , may further be a Reduced Instruction Set Core (RISC) processor, a Complex Instruction Set Computing (CISC) microprocessor, a Microcontroller Unit (MCU), or a CISC-based Central Processing Unit (CPU). The hardware of the processor 12 may be integrated onto a single substrate (e.g., silicon “die”), or distributed among two or more substrates. Furthermore, various functional aspects of the processor 12 may be implemented solely as a software (or firmware) associated with the processor 12 .

A memory can store computer programs or any other sequence of computer readable instructions, or data, such as files, text, numbers, audio and video, as well as any other form of information represented as a string or structure of bits or bytes. The physical means of storing information may be electrostatic, ferroelectric, magnetic, acoustic, optical, chemical, electronic, electrical, or mechanical. A memory may be in the form of an Integrated Circuit (IC, a.k.a. chip or microchip). Alternatively or in addition, a memory may be in the form of a packaged functional assembly of electronic components (module). Such module may be based on a Printed Circuit Board (PCB) such as PC Card according to Personal Computer Memory Card International Association (PCMCIA) PCMCIA 2.0 standard, or a Single In-line Memory Module (SIMM) or a Dual In-line Memory Module (DIMM), standardized under the JEDECJESD-21C standard. Further, a memory may be in the form of a separately rigidly enclosed box such as an external Hard-Disk Drive (HDD), an external Solid State Disk (SSD), or any combination thereof.

Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 12 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The bus 13 carries the data to the main memory 15 a , from which the processor 12 retrieves and executes the instructions. The instructions received by the main memory 15 a may optionally be stored on the storage device 15 c either before or after execution by the processor 12 .

The computer system 11 commonly includes a communication interface 9 coupled to the bus 13 . The communication interface 9 provides a two-way data communication coupling to a network link 8 that is connected to a Local Area Network (LAN) 14 . As a non-limiting example, the communication interface 9 may be a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN. For example, Ethernet-based connection based on IEEE802.3 standard may be used, such as 10/100BaseT, 1000BaseT (gigabit Ethernet), 10 gigabit Ethernet (10GE or 10 GbE or 10 GigE per IEEE Std. 802.3ae-2002as standard), 40 Gigabit Ethernet (40 GbE), or 100 Gigabit Ethernet (100 GbE as per Ethernet standard IEEE P802.3ba). These technologies are described in Cisco Systems, Inc. Publication number 1-587005-001-3 (June 1999), “Internetworking Technologies Handbook”, Chapter 7: “Ethernet Technologies”, pages 7-1 to 7-38, which is incorporated in its entirety for all purposes as if fully set forth herein. In such a case, the communication interface 9 typically includes a LAN transceiver or a modem, such as a Standard Microsystems Corporation (SMSC) LAN91C111 10/100 Ethernet transceiver, described in the Standard Microsystems Corporation (SMSC) data-sheet “LAN91C111 10/100 Non-PCI Ethernet Single Chip MAC+PHY” Data-Sheet, Rev. 15 (Feb. 20, 2004), which is incorporated in its entirety for all purposes as if fully set forth herein.

An Internet Service Provider (ISP) 16 is an organization that provides services for accessing, using, or participating in the Internet 22 . The Internet Service Provider 16 may be organized in various forms, such as commercial, community-owned, non-profit, or otherwise privately owned. Internet services, typically provided by ISPs, include Internet access, Internet transit, domain name registration, web hosting, and collocation. ISPs may engage in peering, where multiple ISPs interconnect at peering points or Internet exchange points (IXs), allowing routing of data between each network, without charging one another for the data transmitted—data that would otherwise have passed through a third upstream ISP, incurring charges from the upstream ISP. ISPs requiring no upstream and having only customers (end customers and/or peer ISPs) are referred to as Tier 1 ISPs.

An arrangement 10 a of a computer system connected to the Internet 22 is shown in FIG. 1 a . A computer system or a workstation 7 includes a main unit box 6 with an enclosed motherboard that has the processor 12 and the memories 15 a , 15 b , and 15 c are mounted. The workstation 7 may include a keyboard 2 (corresponding to the input device 18 ), a printer 4 , a computer mouse 3 (corresponding to the cursor control 18 a ), and a display 5 (corresponding to the display 17 ). FIG. 1 a further illustrates various devices connected via the Internet 22 , such as a client device #1 24 , a client device #2 24 a , a data server #1 23 a , a data server #2 23 b , and the workstation 7 , connected to the Internet 22 over a LAN 14 and via the router or gateway 19 and the ISP 16 .

The client device #1 24 and the client device #2 24 a may communicate over the Internet 22 for exchanging or obtaining data from the data server #1 23 a and the data server #2 23 b . In one example, the servers are HTTP servers, sometimes known as web servers.

The term “computer-readable medium” (or “machine-readable medium”) is used herein to include, but not limited to, any medium or any memory, that participates in providing instructions to a processor, (such as the processor 12 ) for execution, or any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). Such a medium may store computer-executable instructions to be executed by a processing element and/or control logic and data, which is manipulated by a processing element and/or control logic, and may take many forms, including but not limited to, non-volatile medium, volatile medium, and transmission medium. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise the bus 13 . Transmission media may also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications, or other form of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.). Common forms of computer-readable media include a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch-cards, paper-tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer may read.

Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 12 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 11 can receive the data on the telephone line, using an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 13 . The bus 13 carries the data to the main memory 15 a , from which the processor 12 retrieves and executes the instructions. The instructions received by the main memory 15 a may optionally be stored on the storage device 15 c either before or after execution by the processor 12 .

The Internet is a global system of interconnected computer networks that use the standardized Internet Protocol Suite (TCP/IP), including Transmission Control Protocol (TCP) and the Internet Protocol (IP), to serve billions of users worldwide. It is a network of networks that consists of millions of private, public, academic, business, and government networks, of local to global scope, that are linked by a broad array of electronic and optical networking technologies. The Internet carries a vast range of information resources and services, such as the interlinked hypertext documents on the World Wide Web (WWW) and the infrastructure to support electronic mail. The Internet backbone refers to the principal data routes between large, strategically interconnected networks and core routers on the Internet. These data routers are hosted by commercial, government, academic, and other high-capacity network centers, the Internet exchange points and network access points that interchange Internet traffic between the countries, continents and across the oceans of the world. Traffic interchange between Internet service providers (often Tier 1 networks) participating in the Internet backbone exchange traffic by privately negotiated interconnection agreements, primarily governed by the principle of settlement-free peering.

The Internet Protocol is responsible for addressing hosts and routing datagrams (packets) from a source host to the destination host across one or more IP networks. For this purpose, the Internet Protocol defines an addressing system that has two functions: Identifying hosts addresses and providing a logical location service. Each packet is tagged with a header that contains the metadata for the purpose of delivery. This process of tagging is also called encapsulation. IP is a connectionless protocol for use in a packet-switched Link Layer network, and does not need circuit setup prior to transmission. The aspects of guaranteeing delivery, proper sequencing, avoidance of duplicate delivery, and data integrity are addressed by an upper transport layer protocol (e.g., TCP—Transmission Control Protocol and UDP—User Datagram Protocol).

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems, commonly used for communication over the Internet. HTTP is the protocol to exchange or transfer hypertext, which is a structured text that uses logical links (hyperlinks) between nodes containing text. HTTP version 1.1 was standardized as RFC 2616 (June 1999), which was replaced by a set of standards (obsoleting RFC 2616), including RFC 7230—‘HTTP/1.1: Message Syntax and Routing’, RFC 7231—‘HTTP/1.1: Semantics and Content’, RFC 7232—‘HTTP/1.1: Conditional Requests’, RFC 7233—‘HTTP/1.1: Range Requests’, RFC 7234—‘HTTP/1.1: Caching’, and RFC 7235—‘HTTP/1.1: Authentication’. HTTP functions as a request-response protocol in the client-server computing model. A web browser, for example, may be the client and an application running on a computer hosting a website may be the server. The client submits an HTTP request message to the server. The server, which provides resources such as HTML files and other content, or performs other functions on behalf of the client, returns a response message to the client. The response contains completion status information about the request and may further contain a requested content in its message body. A web browser is an example of a User Agent (UA). Other types of user agent include the indexing software used by search providers (web crawlers), voice browsers, mobile apps and other software that accesses, consumes, or displays web content.

User. The term “user” is used herein to include, but not limited to, the principal using a client device or application to interactively retrieve and render resources or resource manifestation, such as a person using a web browser, a person using an e-mail reader, or a person using a display such as the display 17 .

Virtualization. The term virtualization typically refers to the technology that allows for the creation of software-based virtual machines that can run multiple operating systems from a single physical machine. In one example, virtual machines can be used to consolidate the workloads of several under-utilized servers to fewer machines, perhaps a single machine (server consolidation), providing benefits (perceived or real, but often cited by vendors) such as savings on hardware, environmental costs, management, and administration of the server infrastructure. Virtualization scheme allows for the creation of substitutes for real resources, that is, substitutes that have the same functions and external interfaces as their counterparts, but that differ in attributes, such as size, performance, and cost. These substitutes are called virtual resources, and their users are typically unaware of the substitution.

Virtualization is commonly applied to physical hardware resources by combining multiple physical resources into shared pools from which users receive virtual resources. With virtualization, you can make one physical resource look like multiple virtual resources. Virtual resources can have functions or features that are not available in their underlying physical resources. Virtualization can provide the benefits of consolidation to reduce hardware cost, such as to efficiently access and manage resources to reduce operations and systems management costs while maintaining needed capacity, and to have a single server function as multiple virtual servers. In addition, virtualization can provide optimization of workloads, such as to respond dynamically to the application needs of its users, and to increase the use of existing resources by enabling dynamic sharing of resource pools. Further, virtualization may be used for IT flexibility and responsiveness, such as by having a single, consolidated view of, and easy access to, all available resources in the network, regardless of location, and reducing the management of your environment by providing emulation for compatibility and improved interoperability.

Virtual machine (VM). Virtual machine is a representation of a real machine using software that provides an operating environment which can run or host a guest operating system. In one example, a virtual machine may include a self-contained software emulation of a machine, which does not physically exist, but shares resources of an underlying physical machine. Like a physical computer, a virtual machine runs an operating system and applications. Multiple virtual machines can operate concurrently on a single host system. There are different kinds of virtual machines, each with different functions: System virtual machines (also termed full virtualization VMs) provide a substitute for a real machine. They provide functionality needed to execute entire operating systems. A hypervisor uses native execution to share and manage hardware, allowing for multiple environments which are isolated from one another, yet exist on the same physical machine. Modern hypervisors use hardware-assisted virtualization, virtualization-specific hardware, primarily from the host CPUs. Process virtual machines are designed to execute computer programs in a platform-independent environment. Some virtual machines, such as QEMU, are designed to also emulate different architectures and allow execution of software applications and operating systems written for another CPU or architecture. Operating-system-level virtualization allows the resources of a computer to be partitioned via the kernel's support for multiple isolated user space instances, which are usually called containers and may look and feel like real machines to the end users.

Guest Operating System. A guest operating system is an operating system running in a virtual machine environment that would otherwise run directly on a separate physical system. Operating-system-level virtualization, also known as containerization, refers to an operating system feature in which the kernel allows the existence of multiple isolated user-space instances. Such instances, called containers, partitions, Virtualization Engines (VEs) or jails (FreeBSD jail or chroot jail), may look like real computers from the point of view of programs running in them. A computer program running on an ordinary operating system can see all resources (connected devices, files and folders, network shares, CPU power, quantifiable hardware capabilities) of that computer. However, programs running inside a container can only see the container's contents and devices assigned to the container. In addition to isolation mechanisms, the kernel often provides resource-management features to limit the impact of one container's activities on other containers. With operating-system-virtualization, or containerization, it is possible to run programs within containers, to which only parts of these resources are allocated. A program expecting to see the whole computer, once run inside a container, can only see the allocated resources and believes them to be all that is available. Several containers can be created on each operating system, to each of which a subset of the computer's resources is allocated. Each container may contain any number of computer programs. These programs may run concurrently or separately, even interact with each other.

Hypervisor. Hypervisor commonly refers to a thin layer of software that generally provides virtual partitioning capabilities which runs directly on hardware, but underneath higher-level virtualization services. The hypervisor typically manages virtual machines, allowing them to interact directly with the underlying hardware. System virtualization creates many virtual systems within a single physical system. Virtual systems are independent operating environments that use virtual resources. System virtualization can be approached through hardware partitioning or hypervisor technology. Hardware partitioning subdivides a physical server into fractions, each of which can run an operating system. These fractions are typically created with coarse units of allocation, such as whole processors or physical boards. This type of virtualization allows for hardware consolidation, but does not have the full benefits of resource sharing and emulation offered by hypervisors. Hypervisors use a thin layer of code in software or firmware to achieve fine-grained, dynamic resource sharing. Because hypervisors provide the greatest level of flexibility in how virtual resources are defined and managed, they are the primary technology for system virtualization.

Virtual Machine Monitor. A Virtual Machine Monitor (VMM) is computer software, firmware or hardware that creates and runs virtual machines. A computer on which a hypervisor runs one or more virtual machines is called a host machine, and each virtual machine is called a guest machine. The hypervisor presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems. Multiple instances of a variety of operating systems may share the virtualized hardware resources: for example, Linux, Windows, and macOS instances can all run on a single physical x86 machine. This contrasts with operating-system-level virtualization, where all instances (usually called containers) must share a single kernel, though the guest operating systems can differ in user space, such as different Linux distributions with the same kernel. Typically, a VMM refers to a software that runs in a layer between a hypervisor or host operating system and one or more virtual machines that provides the virtual machines abstraction to the guest operating systems. With full virtualization, the VMM exports a virtual machine abstraction identical to the physical machine, so the standard operating system can run just as they would on physical hardware.

Hardware virtualization or platform virtualization refers to the creation of a virtual machine that acts like a real computer with an operating system. Software executed on these virtual machines is separated from the underlying hardware resources. In hardware virtualization, the host machine is the actual machine on which the virtualization takes place, and the guest machine is the virtual machine. The words host and guest are used to distinguish the software that runs on the physical machine from the software that runs on the virtual machine. The software or firmware that creates a virtual machine on the host hardware is called a hypervisor or Virtual Machine Manager. Different types of hardware virtualization include full-virtualization, where almost complete simulation of the actual hardware to allow software, which typically consists of a guest operating system, to run unmodified, and Para-virtualization, where a hardware environment is not simulated; however, the guest programs are executed in their own isolated domains, as if they are running on a separate system. Guest programs need to be specifically modified to run in this environment.

Hardware-assisted virtualization is a way of improving overall efficiency of virtualization. It involves CPUs that provide support for virtualization in hardware, and other hardware components that help improve the performance of a guest environment. Hardware virtualization can be viewed as part of an overall trend in enterprise IT that includes autonomic computing, a scenario in which the IT environment will be able to manage itself based on perceived activity, and utility computing, in which computer processing power is seen as a utility that clients can pay for only as needed. The usual goal of virtualization is to centralize administrative tasks while improving scalability and overall hardware-resource utilization. With virtualization, several operating systems can be run in parallel on a single central processing unit (CPU). This parallelism tends to reduce overhead costs and differs from multitasking, which involves running several programs on the same OS. Using virtualization, an enterprise can better manage updates and rapid changes to the operating system and applications without disrupting the user.

Server Virtualization. Server virtualization is a virtualization technique that involves partitioning a physical server into a number of small, virtual servers with the help of virtualization software. In server virtualization, each virtual server runs multiple operating system instances at the same time. A Virtual Private Server (VPS) is a virtual machine sold as a service by an Internet hosting service, that runs its own copy of an Operating System (OS), and customers may have superuser-level access to that operating system instance, so they can install almost any software that runs on that OS. For many purposes they are functionally equivalent to a dedicated physical server, and being software-defined, are able to be much more easily created and configured. They are typically priced much lower than an equivalent physical server. However, as they share the underlying physical hardware with other VPS's, performance may be lower, depending on the workload of any other executing virtual machines. Dedicated Servers may also be more efficient with CPU dependent processes such as hashing algorithms.

Application Virtualization. Application virtualization is software technology that encapsulates computer programs from the underlying operating system on which it is executed. A fully virtualized application is not installed in the traditional sense, although it is still executed as if it were. The application behaves at runtime like it is directly interfacing with the original operating system and all the resources managed by it, but can be isolated or sandboxed to varying degrees. Application virtualization is layered on top of other virtualization technologies, allowing computing resources to be distributed dynamically in real-time. In this context, the term “virtualization” commonly refers to the artifact being encapsulated (application), which is quite different from its meaning in hardware virtualization, where it refers to the artifact being abstracted (physical hardware).

Network Virtualization. Network Virtualization refers to the process of combining hardware and software network resources to create a single pool of resources that make up a virtual network that can be accessed without regard to the physical component. Network virtualization typically involves combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external virtualization, combining many networks or parts of networks into a virtual unit, or internal virtualization, providing network-like functionality to software containers on a single network server.

Storage Virtualization. Storage virtualization refers to the process of consolidating the physical storage from multiple network storage devices so that it appears to be a single storage unit. Within the context of a storage system, there are two primary types of virtualization that can occur: Block virtualization used in this context refers to the abstraction (separation) of logical storage (partition) from physical storage so that it may be accessed without regard to physical storage or heterogeneous structure. This separation allows the administrators of the storage system greater flexibility in how they manage storage for end users. File virtualization addresses the NAS challenges by eliminating the dependencies between the data accessed at the file level and the location where the files are physically stored. This provides opportunities to optimize storage use and server consolidation and to perform non-disruptive file migrations.

Desktop Virtualization. Desktop virtualization refers to the process of virtualizing desktop computers using virtualization software, such that the desktop computer and the associated operating system and applications are separated from the physical client device that is used to access it. Desktop virtualization is software technology that separates the desktop environment and associated application software from the physical client device that is used to access it.

Desktop virtualization can be used in conjunction with application virtualization and user profile management systems, now termed “user virtualization,” to provide a comprehensive desktop environment management system. In this mode, all the components of the desktop are virtualized, which allows for a highly flexible and much more secure desktop delivery model. In addition, this approach supports a more complete desktop disaster recovery strategy as all components are essentially saved in the data center and backed up through traditional redundant maintenance systems. If a user's device or hardware is lost, the restore is straightforward and simple, because the components will be present at login from another device. In addition, because no data is saved to the user's device, if that device is lost, there is much less chance that any critical data can be retrieved and compromised. Virtual Desktop Infrastructure (VDI)—The practice of hosting a desktop environment within a virtual machine that runs on a centralized or remote server.

An example of a virtualization architecture 900 is shown in FIG. 2 , where three virtual machines are exemplified. A Virtual Machine (VM) #1 910 a provides virtualization for the application 901 a that uses the guest OS 902 a , which in turn interfaces with the virtual hardware 903 a that emulates the actual hardware. Similarly, a Virtual Machine (VM) #2 910 b provides virtualization for the application 901 b that uses the guest OS 902 b , which in turn interfaces with the virtual hardware 903 b that emulates the associated actual hardware, and a Virtual Machine (VM) #3 910 c provides virtualization for the application 901 c that uses the guest OS 902 c , which in turn interfaces with the virtual hardware 903 c that emulates the associated actual hardware. The abstraction layer is provided by VMM 904 , allowing of hardware-independence of operating system and applications, provisioning on any single physical system, and managing the applications and the OSs as a single encapsulated unit.

A hosted architecture 900 a for virtualization is shown in FIG. 2 a , where a wide range of actual host hardware 906 may be used by implementing a host operating system 905 layer between the actual hardware 906 and the VMM 904 . Such configuration relies on the host OS 905 for device support and physical resource management. In contrast, a bare-metal architecture 900 b is shown in FIG. 2 b , where a hypervisor layer (in addition to, or as part of, the VMM 904 ) is used as the first layer, allowing the VMM 904 to have direct access to the hardware resources, hence providing more efficient, and greater scalability, robustness, and performance.

Cloud. The term “Cloud” or “Cloud computing” as used herein is defined as a technology infrastructure facilitating supplement, consumption and delivery of IT services, and generally refers to any group of networked computers capable of delivering computing services (such as computations, applications, data access, and data management and storage resources) to end users. This disclosure does not limit the type (such as public or private) of the cloud as well as the underlying system architecture used by the cloud. The IT services are internet based and may involve elastic provisioning of dynamically scalable and time virtualized resources. Although such virtualization environments can be privately deployed and used within local area or wide area networks owned by an enterprise, a number of “cloud service providers” host virtualization environments accessible through the public internet (the “public cloud”) that is generally open to anyone, or through private IP or other type of network accessible only by entities given access to it (a “private cloud.”). Using a cloud-based control server or using the system above may allow for reduced capital or operational expenditures. The users may further access the system using a web browser regardless of their location or what device they are using, and the virtualization technology allows servers and storage devices to be shared and utilization be increased. Examples of public cloud providers include Amazon AWS, Microsoft Azure and Google GCP. Comparison of service features such as computation, storage, and infrastructure of the three cloud service providers (AWS, Microsoft Azure, GCP) is disclosed in an article entitled: “ Highlight the Features of AWS, GCP and Microsoft Azure that Have an Impact when Choosing a Cloud Service Provider ” by Muhammad Ayoub Kamal, Hafiz Wahab Raza, Muhammad Mansoor Alam, and Mazliham Mohd Su'ud, published January 2020 in ‘International Journal of Recent Technology and Engineering (IJRTE)’ ISSN: 2277-3878, Volume-8 by Blue Eyes Intelligence Engineering & Sciences Publication [DOI:10.35940/ijrte.D8573.018520], which is incorporated in its entirety for all purposes as if fully set forth herein.

The term “Software as a Service (SaaS)” as used herein in this application, is defined as a model of software deployment whereby a provider licenses a Software Application (SA) to customers for use as a service on demand. Similarly, an “Infrastructure as a Service” (IaaS) allows enterprises to access virtualized computing systems through the public Internet. The term “customer” as used herein in this application, is defined as a business entity that is served by an SA, provided on the SaaS platform. A customer may be a person or an organization and may be represented by a user that responsible for the administration of the application in aspects of permissions configuration, user related configuration, and data security policy. The service is supplied and consumed over the Internet, thus eliminating requirements to install and run applications locally on a site of a customer as well as simplifying maintenance and support. Particularly it is advantageous in massive business applications. Licensing is a common form of billing for the service and it is paid periodically. SaaS is becoming ever more common as a form of SA delivery over the Internet and is being facilitated in a technology infrastructure called “Cloud Computing”. In this form of SA delivery, where the SA is controlled by a service provider, a customer may experience stability and data security issues. In many cases, the customer is a business organization that is using the SaaS for business purposes such as business software; hence, stability and data security are primary requirements. As part of a cloud service arrangement, any computer system may also be emulated using software running on a hardware computer system. This virtualization allows for multiple instances of a computer system, each referred to as virtual machine, to run on a single machine. Each virtual machine behaves like a computer system running directly on hardware. It is isolated from the other virtual machines, as would two hardware computers. Each virtual machine comprises an instance of an operating system (the “guest operating system”). There is a host operating system running directly on the hardware that supports the software that emulates the hardware, and the emulation software is referred to as a hypervisor.

The term “cloud-based” generally refers to a hosted service that is remotely located from a data source and configured to receive, store and process data delivered by the data source over a network. Cloud-based systems may be configured to operate as a public cloud-based service, a private cloud-based service or a hybrid cloud-based service. A “public cloud-based service” may include a third-party provider that supplies one or more servers to host multi-tenant services. Examples of a public cloud-based service include Amazon Web Services® (AWS®), Microsoft® Azure™ and Google® Compute Engine™ (GCP) as examples. In contrast, a “private” cloud-based service may include one or more servers that host services provided to a single subscriber (enterprise) and a hybrid cloud-based service may be a combination of certain functionality from a public cloud-based service and a private cloud-based service.

Cloud computing and virtualization is described in a book entitled “ Cloud Computing and Virtualization ” authored by Dac-Nhuong Le (Faculty of Information Technology, Haiphong University, Haiphong, Vietnam), Raghvendra Kumar (Department of Computer Science and Engineering, LNCT, Jabalpur, India), Gia Nhu Nguyen (Graduate School, Duy Tan University, Da Nang, Vietnam), and Jyotir Moy Chatterjee (Department of Computer Science and Engineering at GD-RCET, Bhilai, India), and published 2018 by John Wiley & Sons, Inc. [ISBN 978-1-119-48790-6], which is incorporated in its entirety for all purposes as if fully set forth herein. The book describes the adoption of virtualization in data centers creates the need for a new class of networks designed to support elasticity of resource allocation, increasing mobile workloads and the shift to production of virtual workloads, requiring maximum availability. Building a network that spans both physical servers and virtual machines with consistent capabilities demands a new architectural approach to designing and building the IT infrastructure. Performance, elasticity, and logical addressing structures must be considered as well as the management of the physical and virtual networking infrastructure. Once deployed, a network that is virtualization-ready can offer many revolutionary services over a common shared infrastructure. Virtualization technologies from VMware, Citrix and Microsoft encapsulate existing applications and extract them from the physical hardware. Unlike physical machines, virtual machines are represented by a portable software image, which can be instantiated on physical hardware at a moment's notice. With virtualization, comes elasticity where computer capacity can be scaled up or down on demand by adjusting the number of virtual machines actively executing on a given physical server. Additionally, virtual machines can be migrated while in service from one physical server to another.

Extending this further, virtualization creates “location freedom” enabling virtual machines to become portable across an ever-increasing geographical distance. As cloud architectures and multi-tenancy capabilities continue to develop and mature, there is an economy of scale that can be realized by aggregating resources across applications, business units, and separate corporations to a common shared, yet segmented, infrastructure. Elasticity, mobility, automation, and density of virtual machines demand new network architectures focusing on high performance, addressing portability, and the innate understanding of the virtual machine as the new building block of the data center. Consistent network-supported and virtualization-driven policy and controls are necessary for visibility to virtual machines' state and location as they are created and moved across a virtualized infrastructure.

Virtualization technologies in data center environments are described in a eBook authored by Gustavo Alessandro Andrade Santana and published 2014 by Cisco Systems, Inc. (Cisco Press) [ISBN-13: 978-1-58714-324-3] entitled: “ Data Center Virtualization Fundamentals ”, which is incorporated in its entirety for all purposes as if fully set forth herein. PowerVM technology for virtualization is described in IBM Red Book entitled: “ IBM PowerVM Virtualization—Introduction and Configuration ” published by IBM Corporation June 2013, and virtualization basics is described in a paper by IBM Corporation published 2009 entitled: “ Power Systems—Introduction to virtualization ”, which are both incorporated in their entirety for all purposes as if fully set forth herein.

Server. The Internet architecture employs a client-server model, among other arrangements. The terms ‘server’ or ‘server computer’ relates herein to a device or computer (or a plurality of computers) connected to the Internet and is used for providing facilities or services to other computers or other devices (referred to in this context as ‘clients’) connected to the Internet. A server is commonly a host that has an IP address and executes a ‘server program’, and typically operates as a socket listener. Many servers have dedicated functionality such as web server, Domain Name System (DNS) server (described in RFC 1034 and RFC 1035), Dynamic Host Configuration Protocol (DHCP) server (described in RFC 2131 and RFC 3315), mail server, File Transfer Protocol (FTP) server and database server. Similarly, the term ‘client’ is used herein to include, but not limited to, a program or to a device or a computer (or a series of computers) executing this program, which accesses a server over the Internet for a service or a resource. Clients commonly initiate connections that a server may accept. For non-limiting example, web browsers are clients that connect to web servers for retrieving web pages, and email clients connect to mail storage servers for retrieving mails.

A server device (in server/client architecture) typically offers information resources, services, and applications to clients, using a server dedicated or oriented operating system. A server device may consist of, be based on, include, or be included in the work-station 7 shown in FIG. 2 , the computer system 10 shown in FIG. 1 , or the computer 11 shown in FIG. 1 . Current popular server operating systems are based on Microsoft Windows (by Microsoft Corporation, headquartered in Redmond, Wash., U.S.A.), Unix, and Linux-based solutions, such as the ‘Windows Server 2012’ server operating system, which is a part of the Microsoft ‘Windows Server’ OS family, that was released by Microsoft in 2012. ‘Windows Server 2012’ provides enterprise-class datacenter and hybrid cloud solutions that are simple to deploy, cost-effective, application-specific, and user-centric, and is described in Microsoft publication entitled: “ Inside - Out Windows Server 2012”, by William R. Stanek, published 2013 by Microsoft Press, which is incorporated in its entirety for all purposes as if fully set forth herein.

Unix operating system is widely used in servers. It is a multitasking, multiuser computer operating system that exists in many variants, and is characterized by a modular design that is sometimes called the “Unix philosophy”, meaning the OS provides a set of simple tools, which each performs a limited, well-defined function, with a unified filesystem as the primary means of communication, and a shell scripting and command language to combine the tools to perform complex workflows. Unix was designed to be portable, multi-tasking and multi-user in a time-sharing configuration, and Unix systems are characterized by various concepts: the use of plain text for storing data, a hierarchical file system, treating devices and certain types of Inter-Process Communication (IPC) as files, the use of a large number of software tools, and small programs that can be strung together through a command line interpreter using pipes, as opposed to using a single monolithic program that includes all of the same functionality. Unix operating system consists of many utilities along with the master control program, the kernel. The kernel provides services to start and stop programs, handles the file system and other common “low level” tasks that most programs share, and schedules access to avoid conflicts when programs try to access the same resource, or device simultaneously. To mediate such access, the kernel has special rights, reflected in the division between user-space and kernel-space. Unix is described in a publication entitled: “UNIX Tutorial” by tutorialspoint.com, downloaded on July 2014, which is incorporated in its entirety for all purposes as if fully set forth herein.

Client. The term ‘client’ typically refers to an application (or a device executing the application) used for retrieving or rendering resources, or resource manifestations, such as a web browser, an e-mail reader, or a Usenet reader, while the term ‘server’ typically refers to an application (or a device executing the application) used for supplying resources or resource manifestations, and typically offers (or hosts) various services to other network computers and users. These services are usually provided through ports or numbered access points beyond the server's network address. Each port number is usually associated with a maximum of one running program, which is responsible for handling requests to that port. A daemon, being a user program, can in turn access the local hardware resources of that computer by passing requests to the operating system kernel.

A client device (in server/client architecture) typically receives information resources, services, and applications from servers, and is using a client dedicated or oriented operating system. The client device may consist of, be based on, include, or be included in, the workstation 7 , the computer system 10 or the computer 11 . Current popular client operating systems are based on Microsoft Windows (by Microsoft Corporation, headquartered in Redmond, Wash., U.S.A.), which is a series of graphical interface operating systems developed, marketed, and sold by Microsoft. Microsoft Windows is described in Microsoft publications entitled: “ Windows Internals—Part 1” and “ Windows Internals—Part 2”, by Mark Russinovich, David A. Solomon, and Alex loescu, published by Microsoft Press in 2012, which are both incorporated in their entirety for all purposes as if fully set forth herein. Windows 8 is a personal computer operating system developed by Microsoft as part of Windows NT family of operating systems, that was released for general availability on October 2012, and is described in Microsoft Press 2012 publication entitled: “ Introducing Windows 8 —An Overview for IT Professionals ” by Jerry Honeycutt, which is incorporated in its entirety for all purposes as if fully set forth herein.

Chrome OS is a Linux kernel-based operating system designed by Google Inc. out of Mountain View, Calif., U.S.A., to work primarily with web applications. The user interface takes a minimalist approach and consists almost entirely of just the Google Chrome web browser; since the operating system is aimed at users who spend most of their computer time on the Web, the only “native” applications on Chrome OS are a browser, media player and file manager, and hence the Chrome OS is almost a pure web thin client OS.

The Chrome OS is described as including a three-tier architecture: firmware, browser and window manager, and system-level software and userland services. The firmware contributes to fast boot time by not probing for hardware, such as floppy disk drives, that are no longer common on computers, especially netbooks. The firmware also contributes to security by verifying each step in the boot process and incorporating system recovery. The system-level software includes the Linux kernel that has been patched to improve boot performance. The userland software has been trimmed to essentials, with management by Upstart, which can launch services in parallel, re-spawn crashed jobs, and defer services in the interest of faster booting. The Chrome OS user guide is described in the Samsung Electronics Co., Ltd. presentation entitled: “ Google™ Chrome OS USER GUIDE ” published 2011, which is incorporated in its entirety for all purposes as if fully set forth herein.

RTOS. A Real-Time Operating System (RTOS) is an Operating System (OS) intended to serve real-time applications that process data as it comes in, typically without buffer delays. Processing time requirements (including any OS delay) are typically measured in tenths of seconds or shorter increments of time, and is a time bound system which has well defined fixed time constraints. Processing is commonly to be done within the defined constraints, or the system will fail. They either are event driven or time sharing, where event driven systems switch between tasks based on their priorities while time sharing systems switch the task based on clock interrupts. A key characteristic of an RTOS is the level of its consistency concerning the amount of time it takes to accept and complete an application's task; the variability is jitter. A hard real-time operating system has less jitter than a soft real-time operating system. The chief design goal is not high throughput, but rather a guarantee of a soft or hard performance category. An RTOS that can usually or generally meet a deadline is a soft real-time OS, but if it can meet a deadline deterministically it is a hard real-time OS. An RTOS has an advanced algorithm for scheduling, and includes a scheduler flexibility that enables a wider, computer-system orchestration of process priorities. Key factors in a real-time OS are minimal interrupt latency and minimal thread switching latency; a real-time OS is valued more for how quickly or how predictably it can respond than for the amount of work it can perform in a given period of time.

Common designs of RTOS include event-driven, where tasks are switched only when an event of higher priority needs servicing; called preemptive priority, or priority scheduling, and time-sharing, where task are switched on a regular clocked interrupt, and on events; called round robin. Time sharing designs switch tasks more often than strictly needed, but give smoother multitasking, giving the illusion that a process or user has sole use of a machine. In typical designs, a task has three states: Running (executing on the CPU); Ready (ready to be executed); and Blocked (waiting for an event, I/O for example). Most tasks are blocked or ready most of the time because generally only one task can run at a time per CPU. The number of items in the ready queue can vary greatly, depending on the number of tasks the system needs to perform and the type of scheduler that the system uses. On simpler non-preemptive but still multitasking systems, a task has to give up its time on the CPU to other tasks, which can cause the ready queue to have a greater number of overall tasks in the ready to be executed state (resource starvation).

RTOS concepts and implementations are described in an Application Note No. RES05B00008-0100/Rec. 1.00 published January 2010 by Renesas Technology Corp. entitled: “ R 8 C Family—General RTOS Concepts ”, in JAJA Technology Review article published February 2007 [1535-5535/$32.00] by The Association for Laboratory Automation [doi:10.1016/j.jala.2006.10.016] entitled: “ An Overview of Real - Time Operating Systems ”, and in Chapter 2 entitled: “ Basic Concepts of Real Time Operating Systems ” of a book published 2009 [ISBN—978-1-4020-9435-4] by Springer Science+Business Media B.V. entitled: “ Hardware - Dependent Software—Principles and Practice ”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

QNX. One example of RTOS is QNX, which is a commercial Unix-like real-time operating system, aimed primarily at the embedded systems market. QNX was one of the first commercially successful microkernel operating systems and is used in a variety of devices including cars and mobile phones. As a microkernel-based OS, QNX is based on the idea of running most of the operating system kernel in the form of a number of small tasks, known as Resource Managers. In the case of QNX, the use of a microkernel allows users (developers) to turn off any functionality they do not require without having to change the OS itself; instead, those services will simply not run.

FreeRTOS. FreeRTOS™ is a free and open-source Real-Time Operating system developed by Real Time Engineers Ltd., designed to fit on small embedded systems and implements only a very minimalist set of functions: very basic handle of tasks and memory management, and just sufficient API concerning synchronization. Its features include characteristics such as preemptive tasks, support for multiple microcontroller architectures, a small footprint (4.3 Kbytes on an ARM7 after compilation), written in C, and compiled with various C compilers. It also allows an unlimited number of tasks to run at the same time, and no limitation about their priorities as long as used hardware can afford it.

FreeRTOS™ provides methods for multiple threads or tasks, mutexes, semaphores and software timers. A tick-less mode is provided for low power applications, and thread priorities are supported. Four schemes of memory allocation are provided: allocate only; allocate and free with a very simple, fast, algorithm; a more complex but fast allocate and free algorithm with memory coalescence; and C library allocate and free with some mutual exclusion protection. While the emphasis is on compactness and speed of execution, a command line interface and POSIX-like 10 abstraction add-ons are supported. FreeRTOS™ implements multiple threads by having the host program call a thread tick method at regular short intervals.

The thread tick method switches tasks depending on priority and a round-robin scheduling scheme. The usual interval is 1/1000 of a second to 1/100 of a second, via an interrupt from a hardware timer, but this interval is often changed to suit a particular application. FreeRTOS™ is described in a paper by Nicolas Melot (downloaded July 2015) entitled: “ Study of an operating system: FreeRTOS—Operating systems for embedded devices ”, in a paper (dated Sep. 23, 2013) by Dr. Richard Wall entitled: “ Carebot PIC 32 MX 7 ck implementation of Free RTOS ”, FreeRTOS™ modules are described in web pages entitled: “FreeRTOS™ Modules” published in the www.freertos.org website dated 26 Nov. 2006, and FreeRTOS kernel is described in a paper published 1 Apr. 2007 by Rich Goyette of Carleton University as part of ‘SYSC5701: Operating System Methods for Real-Time Applications’, entitled: “ An Analysis and Description of the Inner Workings of the FreeRTOS Kernel ”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

SafeRTOS. SafeRTOS was constructed as a complementary offering to FreeRTOS, with common functionality but with a uniquely designed safety-critical implementation. When the FreeRTOS functional model was subjected to a full HAZOP, weakness with respect to user misuse and hardware failure within the functional model and API were identified and resolved. Both SafeRTOS and FreeRTOS share the same scheduling algorithm, have similar APIs, and are otherwise very similar, but they were developed with differing objectives. SafeRTOS was developed solely in the C language to meet requirements for certification to IEC61508. SafeRTOS is known for its ability to reside solely in the on-chip read only memory of a microcontroller for standards compliance. When implemented in hardware memory, SafeRTOS code can only be utilized in its original configuration, so certification testing of systems using this OS need not re-test this portion of their designs during the functional safety certification process.

VxWorks. VxWorks is an RTOS developed as proprietary software and designed for use in embedded systems requiring real-time, deterministic performance and, in many cases, safety and security certification, for industries, such as aerospace and defense, medical devices, industrial equipment, robotics, energy, transportation, network infrastructure, automotive, and consumer electronics. VxWorks supports Intel architecture, POWER architecture, and ARM architectures. The VxWorks may be used in multicore asymmetric multiprocessing (AMP), symmetric multiprocessing (SMP), and mixed modes and multi-OS (via Type 1 hypervisor) designs on 32- and 64-bit processors. VxWorks comes with the kernel, middleware, board support packages, Wind River Workbench development suite and complementary third-party software and hardware technologies. In its latest release, VxWorks 7, the RTOS has been re-engineered for modularity and upgradeability so the OS kernel is separate from middleware, applications and other packages. Scalability, security, safety, connectivity, and graphics have been improved to address Internet of Things (IoT) needs.

μC/OS. Micro-Controller Operating Systems (MicroC/OS, stylized as μC/OS) is a real-time operating system (RTOS) that is a priority-based preemptive real-time kernel for microprocessors, written mostly in the programming language C, and is intended for use in embedded systems. MicroC/OS allows defining several functions in C, each of which can execute as an independent thread or task. Each task runs at a different priority, and runs as if it owns the central processing unit (CPU). Lower priority tasks can be preempted by higher priority tasks at any time. Higher priority tasks use operating system (OS) services (such as a delay or event) to allow lower priority tasks to execute. OS services are provided for managing tasks and memory, communicating between tasks, and timing.

In one example, part of, or all of, the steps, methods, or flow charts described herein are executed (independently or in cooperation) by a client device, or any device such as the device 35 shown in FIG. 3 . Alternatively or in addition, part of, or all of, the steps, methods, or flow charts described herein are executed (independently or in cooperation) by a server device, such as server 23 a shown as part on the arrangement 30 shown in FIG. 3 . In one example, a client device (such as the device 35 ) and a server (such as the server 23 a ) cooperatively perform part of, or all of, the steps, methods, or flow charts described herein. For example, lower computing power processor 12 may be used in the device 35 , since the heavy or resourceful computations are performed at a remote server. Such scheme may obviate the need for expensive and resourceful device. In another example, memory resources may be saved at the client device by using data stored at a server. The storage 33 in the device 35 may store the Instructions 37 a and the Operating System 37 b . The device 35 may mainly be used for interfacing the user 36 , while the major storing and processing resources and activities are provided by the server 23 a.

The output component 34 may include a color display for displaying screen elements or for organizing on-screen items and controls for data entry. Further, the device may support the display of split-screen views. The input component 38 may include dedicated hard controls for frequently used/accessed functions (e.g., repeat system message). Many systems used re-configurable keys/buttons whose function change depending on the application. For example, a switch may be used to activate the voice recognition system and it may increase system reliability. The input component 38 and the output component 34 may further cooperate to provide both auditory and visual feedback to confirm driver inputs and availability of the speech command. Further, a strategy to alert drivers through auditory tones/beeps in advance of the presentation of information, and/or changes in display status, may be used. This may limit the need for drivers to continuously monitor the system, or repeat system messages.

The device 35 may serve as a client device and may access data, such as retrieving data from, or sending data to, the server 23 a over the Internet 22 , such as via the ISP 16 as described in FIG. 1 above. The communication with the server 23 a may be via a wireless network 39 , by using the antenna 29 and the wireless transceiver 28 in the device 35 .

A diagrammatic representation of a machine in the example form of the computing device 35 within which a set of instructions, for causing the machine to perform any one or more of the methods discussed herein, may be executed. An example of the device 35 that may be used with any of the steps, methods, or flow-charts herein is schematically described as part of an arrangement 30 shown in FIG. 3 . The components in the device 35 communicate over a bus 32 , which may correspond with the bus 13 in the computer 11 . The computing device 35 may include a mobile phone, a smart phone, a netbook computer, a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer etc., within which a set of instructions, for causing the machine to perform any one or more of the methods discussed herein, may be executed. In alternative embodiments, the machine may be connected (e.g., networked) to other machines in an LAN, an intranet, an extranet, or the Internet. The machine may operate in the capacity of a server machine in client-server network environment. The machine may be a Personal Computer (PC), a Set-Top Box (STB), a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” may also include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

The device 35 may also include an interface bus for facilitating communication from various interface devices (for example, one or more output components 34 , one or more peripheral interfaces, and one or more communication components such as the wireless transceiver 28 ) to the basic configuration via the bus/interface controller that controls the bus 32 . Some of the example output components include a graphics processing unit and an audio processing unit, which may be configured to communicate to various external devices such as a display or speakers via one or more A/V ports. One or more example peripheral interfaces may include a serial interface controller or a parallel interface controller, which may be configured to communicate with external devices such as input components (for example, keyboard, mouse, pen, voice input device, touch input device, etc.) or other peripheral output devices (for example, printer, scanner, etc.) via one or more I/O ports.

The device 35 may be part of, may include, or may be integrated with, a general purpose computing device, arranged in accordance with at least some embodiments described herein. In an example basic configuration, the device 35 may include one or more processors 12 and one or more memories or any other computer readable media. A dedicated memory bus may be used to communicate between the processor 12 and the device memories, such as the ROM 15 b , the main memory 15 a , and a storage 33 . Depending on the desired configuration, the processor 12 may be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 12 may include one or more levels of caching, such as a cache memory, a processor core, and registers. The example processor core may include an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a Digital Signal Processing core (DSP Core), or any combination thereof. An example memory controller may also be used with the processor 12 , or in some implementations, the memory controller may be an internal part of the processor 12 .

Depending on the desired configuration, the device memories may be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. The storage 33 may correspond to the storage device 15 c , and may be part of, may comprise, or may be integrated with the ROM 15 b and the main memory 15 a . The storage 33 may include an operating system 37 c , instruction set 37 b that may include steps or part of, or whole of, the flow-charts described herein. The storage 33 may further include a control module, and program data, which may include path data. Any of the memories or storages of the device 35 may include read-only memory (ROM), such as ROM 15 b , flash memory, Dynamic Random Access Memory (DRAM) such as Synchronous DRAM (SDRAM)), a static memory (e.g., flash memory, Static Random Access Memory (SRAM)) and a data storage device, which communicate with each other via the bus 32 .

The device 35 may have additional features or functionality, and additional interfaces to facilitate communications between the basic configuration shown in FIG. 3 and any desired devices and interfaces. For example, a bus/interface controller may be used to facilitate communications between the basic configuration and one or more data storage devices via a storage interface bus. The data storage devices may be one or more removable storage devices, one or more non-removable storage devices, or a combination thereof. Examples of the removable storage and the non-removable storage devices include magnetic disk devices such as flexible disk drives and hard-disk drives (HDDs), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSDs), and tape drives to name a few. Example computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.

The device 35 may receive inputs from a user 36 via an input component 38 , which may correspond with the input device 18 or cursor control 18 a shown as part of the computer 11 in FIG. 1 , or may correspond with the pointing device 3 or the keyboard 2 shown as part of the computer system 7 in FIG. 1 a . In one example, the input component 38 may be used for receiving instructions from the user 36 . The device 35 notifies or outputs information to the user 36 using an output component 34 , which may correspond to the display 17 shown as part of the computer 11 in FIG. 1 , or may correspond with the printer 4 or the screen 5 shown as part of the computer system 7 in FIG. 1 a . In one example, the output component 34 may be used for displaying guidance to the user 36 .

The interface with the user 36 may be based on the input component 38 and the output component 34 . For example, receiving input (visually or acoustically) from the user 36 via the input component 38 . Similarly, outputting data (visually or acoustically) to the user 36 via the output component 34 . The input component 38 may be a piece of computer hardware equipment used to provide data and control signals to an information processing system such as a computer or information appliance. Such input component 38 may be an integrated or a peripheral input device (e.g., hard/soft keyboard, mouse, resistive or capacitive touch display, etc.). Examples of input components include keyboards, mouse, scanners, digital cameras and joysticks. Input components 38 can be categorized based on the modality of input (e.g., mechanical motion, audio, visual, etc.), whether the input is discrete (e.g. pressing of key) or continuous (e.g., a mouse's position, though digitized into a discrete quantity, is fast enough to be considered continuous), the number of degrees of freedom involved (e.g., two-dimensional traditional mice, or three-dimensional navigators designed for CAD applications). Pointing devices (such as ‘computer mouse’), which are input components used to specify a position in space, can further be classified according to whether the input is direct or indirect. With direct input, the input space coincides with the display space, i.e., pointing is done in the space where visual feedback or the pointer appears. Touchscreens and light pens involve direct input. Examples involving indirect input include the mouse and trackball, and whether the positional information is absolute (e.g., on a touch screen) or relative (e.g., with a mouse that can be lifted and repositioned). Direct input is almost necessarily absolute, but indirect input may be either absolute or relative. For example, digitizing graphics tablets that do not have an embedded screen involve indirect input and sense absolute positions and are often run in an absolute input mode, but they may also be set up to simulate a relative input mode like that of a touchpad, where the stylus or puck can be lifted and repositioned.

In the case of wireless networking, the wireless network 39 may use any type of modulation, such as Amplitude Modulation (AM), a Frequency Modulation (FM), or a Phase Modulation (PM). Further, the wireless network 39 may be a control network (such as ZigBee or Z-Wave), a home network, a WPAN (Wireless Personal Area Network), a WLAN (wireless Local Area Network), a WWAN (Wireless Wide Area Network), or a cellular network. An example of a Bluetooth-based wireless controller that may be included in a wireless transceiver is SPBT2632C1A Bluetooth module available from STMicroelectronics NV and described in the data sheet DocID022930 Rev. 6 dated April 2015 entitled: “ SPBT 2632 C 1 A—Bluetooth® technology class -1 module ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Some embodiments may be used in conjunction with one or more types of wireless communication signals and/or systems, for example, Radio Frequency (RF), Infra-Red (IR), Frequency-Division Multiplexing (FDM), Orthogonal FDM (OFDM), Time-Division Multiplexing (TDM), Time-Division Multiple Access (TDMA), Extended TDMA (E-TDMA), General Packet Radio Service (GPRS), extended GPRS, Code-Division Multiple Access (CDMA), Wideband CDMA (WCDMA), CDMA 2000, single-carrier CDMA, multi-carrier CDMA, Multi-Carrier Modulation (MDM), Discrete Multi-Tone (DMT), Bluetooth®, Global Positioning System (GPS), Wi-Fi, Wi-Max, ZigBee™, Ultra-Wideband (UWB), Global System for Mobile communication (GSM), 2G, 2.5G, 3G, 3.5G, Enhanced Data rates for GSM Evolution (EDGE), or the like. Further, a wireless communication may be based on, or may be compatible with, wireless technologies that are described in Chapter 20 : “Wireless Technologies ” of the publication number 1-587005-001-3 by Cisco Systems, Inc. (July 1999) entitled: “ Internetworking Technologies Handbook ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Alternatively or in addition, the networking or the communication with the of the wireless-capable device 35 with the server 23 a over the wireless network 39 may be using, may be according to, may be compatible with, or may be based on, Near Field Communication (NFC) using passive or active communication mode, and may use the 13.56 MHz frequency band, and data rate may be 106 Kb/s, 212 Kb/s, or 424 Kb/s, and the modulation may be Amplitude-Shift-Keying (ASK), and may be according to, may be compatible with, or based on, ISO/IEC 18092, ECMA-340, ISO/IEC 21481, or ECMA-352. In such a case, the wireless transceiver 28 may be an NFC transceiver and the respective antenna 29 may be an NFC antenna.

Alternatively or in addition, the networking or the communication with the of the wireless-capable device 35 with the server 23 a over the wireless network 39 may be using, may be according to, may be compatible with, or may be based on, a Wireless Personal Area Network (WPAN) that may be according to, may be compatible with, or based on, Bluetooth™ or IEEE 802.15.1-2005 standards, and the wireless transceiver 28 may be a WPAN modem, and the respective antenna 29 may be a WPAN antenna. The WPAN may be a wireless control network according to, may be compatible with, or based on, ZigBee™ or Z-Wave™ standards, such as IEEE 802.15.4-2003.

Alternatively or in addition, the networking or the communication with the of the wireless-capable device 35 with the server 23 a over the wireless network 39 may be using, may be according to, may be compatible with, or may be based on, a Wireless Local Area Network (WLAN) that may be according to, may be compatible with, or based on, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, or IEEE 802.11ac standards, and the wireless transceiver 28 may be a WLAN modem, and the respective antenna 29 may be a WLAN antenna.

Alternatively or in addition, the networking or the communication with the of the wireless-capable device 35 with the server 23 a over the wireless network 39 may be using, may be according to, may be compatible with, or may be based on, a wireless broadband network or a Wireless Wide Area Network (WWAN), and the wireless transceiver 28 may be a WWAN modem, and the respective antenna 29 may be a WWAN antenna. The WWAN may be a WiMAX network such as according to, may be compatible with, or based on, IEEE 802.16-2009, and the wireless transceiver 28 may be a WiMAX modem, and the respective antenna 29 may be a WiMAX antenna. Alternatively or in addition, the WWAN may be a cellular telephone network and the wireless transceiver 28 may be a cellular modem, and the respective antenna 29 may be a cellular antenna. The WWAN may be a Third Generation (3G) network and may use UMTS W-CDMA, UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, or GSM EDGE-Evolution. The cellular telephone network may be a Fourth Generation (4G) network and may use HSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be based on, or may be compatible with, IEEE 802.20-2008. Alternatively or in addition, the WWAN may be a satellite network, and the wireless transceiver 28 may be a satellite modem, and the respective antenna 29 may be a satellite antenna.

Alternatively or in addition, the networking or the communication with the of the wireless-capable device 35 with the server 23 a over the wireless network 39 may be using, may be according to, may be compatible with, or may be based on, a licensed or an unlicensed radio frequency band, such as the Industrial, Scientific and Medical (ISM) radio band. For example, an unlicensed radio frequency band may be used that may be about 60 GHz, may be based on beamforming, and may support a data rate of above 7 Gb/s, such as according to, may be compatible with, or based on, WiGig™, IEEE 802.11ad, WirelessHD™ or IEEE 802.15.3c-2009, and may be operative to carry uncompressed video data, and may be according to, may be compatible with, or based on, WHDI™. Alternatively or in addition, the wireless network may use a white space spectrum that may be an analog television channel consisting of a 6 MHz, 7 MHz or 8 MHz frequency band, and allocated in the 54-806 MHz band. The wireless network may be operative for channel bonding, and may use two or more analog television channels, and may be based on Wireless Regional Area Network (WRAN) standard using OFDMA modulation. Further, the wireless communication may be based on geographically-based cognitive radio, and may be according to, may be compatible with, or based on, IEEE 802.22 or IEEE 802.11af standards. Real-Time Clock (RTC) ICs measure time even when the power of the main device is off. During these times, RTC ICs draw power from an auxiliary battery or supercapacitor. Most modern RTC ICs reduce package pin count by supporting a serial interface. An example of an RTC IC is model No. DS1339A available from Maxim Integrated Products, Inc. (Headquartered in San Jose, Calif., U.S.A.), described in a data sheet No. 19-6425; Rev 2; January 2015 (2015) by Maxim Integrated Products, Inc. entitled: “ DS 1339 A—Low - Current, PC, Serial Real - Time Clock ”, which is incorporated in its entirety for all purposes as if fully set forth herein, and may be used as described in a tutorial 5791 (dated Mar. 28, 2014) by Maxim Integrated Products, Inc. entitled: “ Tips for Writing Bulletproof Real - Time Clock Control Code ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Smartphone. A mobile phone (also known as a cellular phone, cell phone, smartphone, or hand phone) is a device which can make and receive telephone calls over a radio link whilst moving around a wide geographic area, by connecting to a cellular network provided by a mobile network operator. The calls are to and from the public telephone network, which includes other mobiles and fixed-line phones across the world. The Smartphones are typically hand-held and may combine the functions of a personal digital assistant (PDA), and may serve as portable media players and camera phones with high-resolution touch-screens, web browsers that can access, and properly display, standard web pages rather than just mobile-optimized sites, GPS navigation, Wi-Fi, and mobile broadband access. In addition to telephony, the Smartphones may support a wide variety of other services such as text messaging, MMS, email, Internet access, short-range wireless communications (infrared, Bluetooth), business applications, gaming and photography.

An example of a contemporary smartphone is model iPhone 12 Pro Max available from Apple Inc., headquartered in Cupertino, Calif., U.S.A. and described in iPhone 12 Pro Max technical specification and in a web-page by Apple Inc. entitled: “About iOS 14 Updates” (both retrieved November 2020 from www.apple.com), which are both incorporated in their entirety for all purposes as if fully set forth herein. Another example of a smartphone is Samsung Galaxy S20 available from Samsung Electronics headquartered in Suwon, South-Korea, described in a document number UNL_STR_G981U_G986U_G988U_EN_UM_TN_TAW_021220_FINAL entitled: “ Galaxy S 20 /S 20 +/S 20 Untra 5 G—User manual ” (retrieved November 2020 from www.samsung.com), which is incorporated in its entirety for all purposes as if fully set forth herein.

Android is an open source and Linux-based mobile operating system (OS) based on the Linux kernel that is currently offered by Google. With a user interface based on direct manipulation, Android is designed primarily for touchscreen mobile devices such as smartphones and tablet computers, with specialized user interfaces for televisions (Android TV), cars (Android Auto), and wrist watches (Android Wear). The OS uses touch inputs that loosely correspond to real-world actions, such as swiping, tapping, pinching, and reverse pinching to manipulate on-screen objects, and a virtual keyboard. Despite being primarily designed for touchscreen input, it also has been used in game consoles, digital cameras, and other electronics. The response to user input is designed to be immediate and provides a fluid touch interface, often using the vibration capabilities of the device to provide haptic feedback to the user. Internal hardware such as accelerometers, gyroscopes and proximity sensors are used by some applications to respond to additional user actions, for example, adjusting the screen from portrait to landscape depending on how the device is oriented, or allowing the user to steer a vehicle in a racing game by rotating the device by simulating control of a steering wheel.

Android devices boot to the homescreen, the primary navigation and information point on the device, which is similar to the desktop found on PCs. Android homescreens are typically made up of app icons and widgets; app icons launch the associated app, whereas widgets display live, auto-updating content such as the weather forecast, the user's email inbox, or a news ticker directly on the homescreen. A homescreen may be made up of several pages that the user can swipe back and forth between, though Android's homescreen interface is heavily customizable, allowing the user to adjust the look and feel of the device to their tastes. Third-party apps available on Google Play and other app stores can extensively re-theme the homescreen, and even mimic the look of other operating systems, such as Windows Phone. The Android OS is described in a publication entitled: “ Android Tutorial ”, downloaded from tutorialspoint.com on July 2014, which is incorporated in its entirety for all purposes as if fully set forth herein.

iOS (previously iPhone OS) from Apple Inc. (headquartered in Cupertino, Calif., U.S.A.) is a mobile operating system distributed exclusively for Apple hardware. The user interface of the iOS is based on the concept of direct manipulation, using multi-touch gestures. Interface control elements consist of sliders, switches, and buttons. Interaction with the OS includes gestures such as swipe, tap, pinch, and reverse pinch, all of which have specific definitions within the context of the iOS operating system and its multi-touch interface. Internal accelerometers are used by some applications to respond to shaking the device (one common result is the undo command) or rotating it in three dimensions (one common result is switching from portrait to landscape mode). The iOS OS is described in a publication entitled: “ IOS Tutorial ”, downloaded from tutorialspoint.com on July 2014, which is incorporated in its entirety for all purposes as if fully set forth herein.

Tablet. A tablet computer, commonly referred to as ‘tablet’, is a mobile device, typically with a mobile operating system and touchscreen display processing circuitry, and a rechargeable battery in a single, thin and flat package. Modern tablets largely resemble modern smartphones and are used for personal, educational and workplace applications, and the only differences being that tablets are relatively larger than smartphones, with screens 7 inches (18 cm) or larger, measured diagonally, and may not support access to a cellular network. The touchscreen display is typically operated by gestures executed by finger or digital pen (stylus), instead of the mouse, trackpad, and keyboard of larger computers. Portable computers can be classified according to the presence and appearance of physical keyboards. Two species of tablet, the slate and booklet, do not have physical keyboards and usually accept text and other input by use of a virtual keyboard shown on their touchscreen displays. To compensate for their lack of a physical keyboard, most tablets can connect to independent physical keyboards by Bluetooth or USB.

The size of a slate shaped tablets varies, but commonly slates begin at 6 inches (approximately 15 cm). Some models in the larger than 10-inch (25 cm). Mini tablets are smaller and weigh less than slates, with typical screen sizes between 7-8 inches (18-20 cm). Smartphones and tablets are similar devices, differentiated by the former typically having smaller screens and most tablets lacking cellular network capability.

Two major architectures dominate the tablet market, ARM Holdings' ARM architecture and Intel's and AMD's x86. A key component among tablet computers is touch input on a touchscreen display. This allows the user to navigate easily and type with a virtual keyboard on the screen or press other icons on the screen to open apps or files. The system must respond to on-screen touches rather than clicks of a keyboard or mouse. This operation makes precise use of our eye-hand coordination. Touchscreens usually come in one of two forms—resistive and capacitive. Resistive touchscreens are passive and respond to pressure on the screen. They allow a high level of precision, useful in emulating a pointer (as is common in tablet computers) but may require calibration. Because of the high resolution, a stylus or fingernail is often used. Stylus-oriented systems are less suited to multi-touch. Capacitive touchscreens tend to be less accurate, but more responsive than resistive devices. Because they require a conductive material, such as a fingertip, for input, they are not common among stylus-oriented devices but are prominent on consumer devices. Most finger-driven capacitive screens do not currently support pressure input, but some tablets use a pressure-sensitive stylus or active pen. Some tablets can recognize individual palms, while some professional-grade tablets use pressure-sensitive films, such as those on graphics tablets. Some capacitive touch-screens can detect the size of the touched area and the pressure used.

Operating system. A mobile operating system (also referred to as mobile OS), is an operating system that operates a smartphone, tablet, PDA, or another mobile device. Modern mobile operating systems combine the features of a personal computer operating system with other features, including a touchscreen, cellular, Bluetooth, Wi-Fi, GPS mobile navigation, camera, video camera, speech recognition, voice recorder, music player, near field communication and infrared blaster. Currently, the popular mobile OSs include Android, Symbian, Apple iOS, BlackBerry, MeeGo, Windows Phone, and Bada. Mobile devices with mobile communications capabilities (e.g. smartphones) typically contain two mobile operating systems: a main user-facing software platform is supplemented by a second low-level proprietary real-time operating system that operates the radio and other hardware.

Android is a Linux-based, open source mobile operating system (OS) based on the Linux kernel that is currently offered by Google. With a user interface based on direct manipulation, Android is designed primarily for touchscreen mobile devices such as smartphones and tablet computers with specialized user interfaces for televisions (Android TV), cars (Android Auto), and wrist watches (Android Wear). The OS uses touch inputs that loosely correspond to real-world actions, such as swiping, tapping, pinching, and reverse pinching to manipulate on-screen objects, and a virtual keyboard. Despite being primarily designed for touchscreen input, it also has been used in game consoles, digital cameras, and other electronics. The response to user input is designed to be immediate and provides a fluid touch interface, often using the vibration capabilities of the device to provide haptic feedback to the user. Internal hardware such as accelerometers, gyroscopes and proximity sensors are used by some applications to respond to additional user actions. For example, adjusting the screen from portrait to landscape depending on the device orientation, or allowing the user to steer a vehicle in a racing game by rotating the device, a process that simulates control of a steering wheel.

Android devices boot to the homescreen, the primary navigation and information point on the device, which is similar to the desktop found on PCs. The homescreens on Android are typically made up of app icons and widgets. App icons launch the associated app, whereas widgets display live, auto-updating content such as the weather forecast, the user's email inbox, or a news ticker directly on the homescreen. A homescreen may be made up of several pages that the user can swipe back and forth between pages. A heavily-customizable Android homescreen interface allows the user to adjust the look and feel of the device to their liking. Third-party apps available on Google Play and other app stores can extensively re-theme the homescreen, and even mimic the look of other operating systems, such as Windows Phone. The Android OS is described in a publication entitled: “ Android Tutorial ”, downloaded from tutorialspoint.com on July 2014, which is incorporated in its entirety for all purposes as if fully set forth herein.

iOS (previously iPhone OS) from Apple Inc. (headquartered in Cupertino, Calif., U.S.A.) is a mobile operating system distributed exclusively for Apple hardware. The user interface of the iOS is based on the concept of direct manipulation, using multi-touch gestures. Interface control elements consist of sliders, switches, and buttons. Interaction with the OS includes gestures such as swipe, tap, pinch, and reverse pinch, all of which have specific definitions within the context of the iOS operating system and its multi-touch interface. Internal accelerometers are used by some applications to respond to shaking the device (one common result is the undo command), or rotating it in three dimensions (one common result is switching from portrait to landscape mode). The iOS is described in a publication entitled: “ IOS Tutorial ”, downloaded from tutorialspoint.com on July 2014, which is incorporated in its entirety for all purposes as if fully set forth herein.

Wireless. Any embodiment herein may be used in conjunction with one or more types of wireless communication signals and/or systems, for example, Radio Frequency (RF), Infra-Red (IR), Frequency-Division Multiplexing (FDM), Orthogonal FDM (OFDM), Time-Division Multiplexing (TDM), Time-Division Multiple Access (TDMA), Extended TDMA (E-TDMA), General Packet Radio Service (GPRS), extended GPRS, Code-Division Multiple Access (CDMA), Wideband CDMA (WCDMA), CDMA 2000, single-carrier CDMA, multi-carrier CDMA, Multi-Carrier Modulation (MDM), Discrete Multi-Tone (DMT), Bluetooth®, Global Positioning System (GPS), Wi-Fi, Wi-Max, ZigBee™, Ultra-Wideband (UWB), Global System for Mobile communication (GSM), Second Generation (2G), 2.5G, Third Generation (3G), 3.5G, Enhanced Data rates for GSM Evolution (EDGE), Fourth Generation (4G), Fifth Generation (5G), or the like. Any wireless network or wireless connection herein may be operating substantially in accordance with existing IEEE 802.11, 802.11a, 802.11b, 802.11g, 802.11k, 802.11n, 802.11r, 802.16, 802.16d, 802.16e, 802.20, 802.21 standards and/or future versions and/or derivatives of the above standards. Further, a network element (or a device) herein may consist of, be part of, or include, a cellular radio-telephone communication system, a cellular telephone, a wireless telephone, a Personal Communication Systems (PCS) device, a PDA device that incorporates a wireless communication device, or a mobile/portable Global Positioning System (GPS) device. Further, a wireless communication may be based on wireless technologies that are described in Chapter 20 : “Wireless Technologies ” of the publication number 1-587005-001-3 by Cisco Systems, Inc. (July 2099) entitled: “ Internetworking Technologies Handbook ”, which is incorporated in its entirety for all purposes as if fully set forth herein. Wireless technologies and networks are further described in a book published 2005 by Pearson Education, Inc. William Stallings [ISBN: 0-13-191835-4] entitled: “ Wireless Communications and Networks—second Edition ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Wireless networking typically employs an antenna (a.k.a. aerial), which is an electrical device that converts electric power into radio waves, and vice versa, connected to a wireless radio transceiver. In transmission, a radio transmitter supplies an electric current oscillating at radio frequency to the antenna terminals, and the antenna radiates the energy from the current as electromagnetic waves (radio waves). In reception, an antenna intercepts some of the power of an electromagnetic wave in order to produce a low voltage at its terminals that is applied to a receiver to be amplified. Typically an antenna consists of an arrangement of metallic conductors (elements), electrically connected (often through a transmission line) to the receiver or transmitter. An oscillating current of electrons forced through the antenna by a transmitter will create an oscillating magnetic field around the antenna elements, while the charge of the electrons also creates an oscillating electric field along the elements. These time-varying fields radiate away from the antenna into space as a moving transverse electromagnetic field wave. Conversely, during reception, the oscillating electric and magnetic fields of an incoming radio wave exert force on the electrons in the antenna elements, causing them to move back and forth, creating oscillating currents in the antenna. Antennas can be designed to transmit and receive radio waves in all horizontal directions equally (omnidirectional antennas), or preferentially in a particular direction (directional or high gain antennas). In the latter case, an antenna may also include additional elements or surfaces with no electrical connection to the transmitter or receiver, such as parasitic elements, parabolic reflectors or horns, which serve to direct the radio waves into a beam or other desired radiation pattern.

ISM. The Industrial, Scientific and Medical (ISM) radio bands are radio bands (portions of the radio spectrum) reserved internationally for the use of radio frequency (RF) energy for industrial, scientific and medical purposes other than telecommunications. In general, communications equipment operating in these bands must tolerate any interference generated by ISM equipment, and users have no regulatory protection from ISM device operation. The ISM bands are defined by the ITU-R in 5.138, 5.150, and 5.280 of the Radio Regulations. Individual countries use of the bands designated in these sections may differ due to variations in national radio regulations. Because communication devices using the ISM bands must tolerate any interference from ISM equipment, unlicensed operations are typically permitted to use these bands, since unlicensed operation typically needs to be tolerant of interference from other devices anyway. The ISM bands share allocations with unlicensed and licensed operations; however, due to the high likelihood of harmful interference, licensed use of the bands is typically low. In the United States, uses of the ISM bands are governed by Part 18 of the Federal Communications Commission (FCC) rules, while Part 15 contains the rules for unlicensed communication devices, even those that share ISM frequencies. In Europe, the ETSI is responsible for governing ISM bands.

Commonly used ISM bands include a 2.45 GHz band (also known as 2.4 GHz band) that includes the frequency band between 2.400 GHz and 2.500 GHz, a 5.8 GHz band that includes the frequency band 5.725-5.875 GHz, a 24 GHz band that includes the frequency band 24.000-24.250 GHz, a 61 GHz band that includes the frequency band 61.000-61.500 GHz, a 122 GHz band that includes the frequency band 122.000-123.000 GHz, and a 244 GHz band that includes the frequency band 244.000-246.000 GHz.

ZigBee. ZigBee is a standard for a suite of high-level communication protocols using small, low-power digital radios based on an IEEE 802 standard for Personal Area Network (PAN). Applications include wireless light switches, electrical meters with in-home-displays, and other consumer and industrial equipment that require a short-range wireless transfer of data at relatively low rates. The technology defined by the ZigBee specification is intended to be simpler and less expensive than other WPANs, such as Bluetooth. ZigBee is targeted at Radio-Frequency (RF) applications that require a low data rate, long battery life, and secure networking. ZigBee has a defined rate of 250 kbps suited for periodic or intermittent data or a single signal transmission from a sensor or input device.

ZigBee builds upon the physical layer and medium access control defined in IEEE standard 802.15.4 (2003 version) for low-rate WPANs. The specification further discloses four main components: network layer, application layer, ZigBee Device Objects (ZDOs), and manufacturer-defined application objects, which allow for customization and favor total integration. The ZDOs are responsible for a number of tasks, which include keeping of device roles, management of requests to join a network, device discovery, and security. Because ZigBee nodes can go from a sleep to active mode in 30 ms or less, the latency can be low and devices can be responsive, particularly compared to Bluetooth wake-up delays, which are typically around three seconds. ZigBee nodes can sleep most of the time, thus the average power consumption can be lower, resulting in longer battery life.

There are three defined types of ZigBee devices: ZigBee Coordinator (ZC), ZigBee Router (ZR), and ZigBee End Device (ZED). ZigBee Coordinator (ZC) is the most capable device and forms the root of the network tree and might bridge to other networks. There is exactly one defined ZigBee coordinator in each network, since it is the device that started the network originally. It is able to store information about the network, including acting as the Trust Center & repository for security keys. ZigBee Router (ZR) may be running an application function as well as may be acting as an intermediate router, passing on data from other devices. ZigBee End Device (ZED) contains functionality to talk to a parent node (either the coordinator or a router). This relationship allows the node to be asleep a significant amount of the time, thereby giving long battery life. A ZED requires the least amount of memory, and therefore can be less expensive to manufacture than a ZR or ZC.

The protocols build on recent algorithmic research (Ad-hoc On-demand Distance Vector, neuRFon) to automatically construct a low-speed ad-hoc network of nodes. In most large network instances, the network will be a cluster of clusters. It can also form a mesh or a single cluster. The current ZigBee protocols support beacon and non-beacon enabled networks. In non-beacon-enabled networks, an unslotted CSMA/CA channel access mechanism is used. In this type of network, ZigBee Routers typically have their receivers continuously active, requiring a more robust power supply. However, this allows for heterogeneous networks in which some devices receive continuously, while others only transmit when an external stimulus is detected.

In beacon-enabled networks, the special network nodes called ZigBee Routers transmit periodic beacons to confirm their presence to other network nodes. Nodes may sleep between the beacons, thus lowering their duty cycle and extending their battery life. Beacon intervals depend on the data rate; they may range from 15.36 milliseconds to 251.65824 seconds at 250 Kbit/s, from 24 milliseconds to 393.216 seconds at 40 Kbit/s, and from 48 milliseconds to 786.432 seconds at 20 Kbit/s. In general, the ZigBee protocols minimize the time the radio is on to reduce power consumption. In beaconing networks, nodes only need to be active while a beacon is being transmitted. In non-beacon-enabled networks, power consumption is decidedly asymmetrical: some devices are always active while others spend most of their time sleeping.

Except for the Smart Energy Profile 2.0, current ZigBee devices conform to the IEEE 802.15.4-2003 Low-Rate Wireless Personal Area Network (LR-WPAN) standard. The standard specifies the lower protocol layers—the PHYsical layer (PHY), and the Media Access Control (MAC) portion of the Data Link Layer (DLL). The basic channel access mode is “Carrier Sense, Multiple Access/Collision Avoidance” (CSMA/CA), that is, the nodes talk in the same way that people converse; they briefly check to see that no one is talking before they start. There are three notable exceptions to the use of CSMA. Beacons are sent on a fixed time schedule, and do not use CSMA. Message acknowledgments also do not use CSMA. Finally, devices in Beacon Oriented networks that have low latency real-time requirement, may also use Guaranteed Time Slots (GTS), which by definition do not use CSMA.

Z-Wave. Z-Wave is a wireless communications protocol by the Z-Wave Alliance (http://www.z-wave.com) designed for home automation, specifically for remote control applications in residential and light commercial environments. The technology uses a low-power RF radio embedded or retrofitted into home electronics devices and systems, such as lighting, home access control, entertainment systems and household appliances. Z-Wave communicates using a low-power wireless technology designed specifically for remote control applications. Z-Wave operates in the sub-gigahertz frequency range, around 900 MHz. This band competes with some cordless telephones and other consumer electronics devices, but avoids interference with WiFi and other systems that operate on the crowded 2.4 GHz band. Z-Wave is designed to be easily embedded in consumer electronics products, including battery-operated devices such as remote controls, smoke alarms, and security sensors.

Z-Wave is a mesh networking technology where each node or device on the network is capable of sending and receiving control commands through walls or floors, and use intermediate nodes to route around household obstacles or radio dead spots that might occur in the home. Z-Wave devices can work individually or in groups, and can be programmed into scenes or events that trigger multiple devices, either automatically or via remote control. The Z-wave radio specifications include bandwidth of 9,600 bit/s or 40 Kbit/s, fully interoperable, GFSK modulation, and a range of approximately 100 feet (or 30 meters) assuming “open air” conditions, with reduced range indoors depending on building materials, etc. The Z-Wave radio uses the 900 MHz ISM band: 908.42 MHz (United States); 868.42 MHz (Europe); 919.82 MHz (Hong Kong); and 921.42 MHz (Australia/New Zealand).

Z-Wave uses a source-routed mesh network topology and has one or more master controllers that control routing and security. The devices can communicate to another by using intermediate nodes to actively route around, and circumvent household obstacles or radio dead spots that might occur. A message from node A to node C can be successfully delivered even if the two nodes are not within range, providing that a third node B can communicate with nodes A and C. If the preferred route is unavailable, the message originator will attempt other routes until a path is found to the “C” node. Therefore, a Z-Wave network can span much farther than the radio range of a single unit; however, with several of these hops, a delay may be introduced between the control command and the desired result. In order for Z-Wave units to be able to route unsolicited messages, they cannot be in sleep mode. Therefore, most battery-operated devices are not designed as repeater units. A Z-Wave network can consist of up to 232 devices with the option of bridging networks if more devices are required.

WWAN. Any wireless network herein may be a Wireless Wide Area Network (WWAN) such as a wireless broadband network, and the WWAN port may be an antenna and the WWAN transceiver may be a wireless modem. The wireless network may be a satellite network, the antenna may be a satellite antenna, and the wireless modem may be a satellite modem. The wireless network may be a WiMAX network such as according to, compatible with, or based on, IEEE 802.16-2009, the antenna may be a WiMAX antenna, and the wireless modem may be a WiMAX modem. The wireless network may be a cellular telephone network, the antenna may be a cellular antenna, and the wireless modem may be a cellular modem. The cellular telephone network may be a Third Generation (3G) network, and may use UMTS W-CDMA, UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, or GSM EDGE-Evolution. The cellular telephone network may be a Fourth Generation (4G) network and may use or be compatible with HSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be compatible with, or based on, IEEE 802.20-2008.

WLAN. Wireless Local Area Network (WLAN), is a popular wireless technology that makes use of the Industrial, Scientific and Medical (ISM) frequency spectrum. In the US, three of the bands within the ISM spectrum are the A band, 902-928 MHz; the B band, 2.4-2.484 GHz (a.k.a. 2.4 GHz); and the C band, 5.725-5.875 GHz (a.k.a. 5 GHz). Overlapping and/or similar bands are used in different regions such as Europe and Japan. In order to allow interoperability between equipment manufactured by different vendors, few WLAN standards have evolved, as part of the IEEE 802.11 standard group, branded as WiFi (www.wi-fi.org). IEEE 802.11b describes a communication using the 2.4 GHz frequency band and supporting communication rate of 11 Mb/s, IEEE 802.11a uses the 5 GHz frequency band to carry 54 MB/s and IEEE 802.11g uses the 2.4 GHz band to support 54 Mb/s. The WiFi technology is further described in a publication entitled: “WiFi Technology” by Telecom Regulatory Authority, published on July 2003, which is incorporated in its entirety for all purposes as if fully set forth herein. The IEEE 802 defines an ad-hoc connection between two or more devices without using a wireless access point: the devices communicate directly when in range. An ad hoc network offers peer-to-peer layout and is commonly used in situations such as a quick data exchange or a multiplayer LAN game, because the setup is easy and an access point is not required.

A node/client with a WLAN interface is commonly referred to as STA (Wireless Station/Wireless client). The STA functionality may be embedded as part of the data unit, or alternatively be a dedicated unit, referred to as bridge, coupled to the data unit. While STAs may communicate without any additional hardware (ad-hoc mode), such network usually involves Wireless Access Point (a.k.a. WAP or AP) as a mediation device. The WAP implements the Basic Stations Set (BSS) and/or ad-hoc mode based on Independent BSS (IBSS). STA, client, bridge and WAP will be collectively referred to hereon as WLAN unit. Bandwidth allocation for IEEE 802.11g wireless in the U.S. allows multiple communication sessions to take place simultaneously, where eleven overlapping channels are defined spaced 5 MHz apart, spanning from 2412 MHz as the center frequency for channel number 1, via channel 2 centered at 2417 MHz and 2457 MHz as the center frequency for channel number 10, up to channel 11 centered at 2462 MHz. Each channel bandwidth is 22 MHz, symmetrically (+/−11 MHz) located around the center frequency. In the transmission path, first the baseband signal (IF) is generated based on the data to be transmitted, using 256 QAM (Quadrature Amplitude Modulation) based OFDM (Orthogonal Frequency Division Multiplexing) modulation technique, resulting a 22 MHz (single channel wide) frequency band signal. The signal is then up converted to the 2.4 GHz (RF) and placed in the center frequency of required channel, and transmitted to the air via the antenna. Similarly, the receiving path comprises a received channel in the RF spectrum, down converted to the baseband (IF) wherein the data is then extracted.

In order to support multiple devices and using a permanent solution, a Wireless Access Point (WAP) is typically used. A Wireless Access Point (WAP, or Access Point—AP) is a device that allows wireless devices to connect to a wired network using Wi-Fi, or related standards. The WAP usually connects to a router (via a wired network) as a standalone device, but can also be an integral component of the router itself. Using Wireless Access Point (AP) allows users to add devices that access the network with little or no cables. A WAP normally connects directly to a wired Ethernet connection, and the AP then provides wireless connections using radio frequency links for other devices to utilize that wired connection. Most APs support the connection of multiple wireless devices to one wired connection. Wireless access typically involves special security considerations, since any device within a range of the WAP can attach to the network. The most common solution is wireless traffic encryption. Modern access points come with built-in encryption such as Wired Equivalent Privacy (WEP) and Wi-Fi Protected Access (WPA), typically used with a password or a passphrase. Authentication in general, and a WAP authentication in particular, is used as the basis for authorization, which determines whether a privilege may be granted to a particular user or process, privacy, which keeps information from becoming known to non-participants, and non-repudiation, which is the inability to deny having done something that was authorized to be done based on the authentication. An authentication in general, and a WAP authentication in particular, may use an authentication server that provides a network service that applications may use to authenticate the credentials, usually account names and passwords of their users. When a client submits a valid set of credentials, it receives a cryptographic ticket that it can subsequently be used to access various services. Authentication algorithms include passwords, Kerberos, and public key encryption.

Prior art technologies for data networking may be based on single carrier modulation techniques, such as AM (Amplitude Modulation), FM (Frequency Modulation), and PM (Phase Modulation), as well as bit encoding techniques such as QAM (Quadrature Amplitude Modulation) and QPSK (Quadrature Phase Shift Keying). Spread spectrum technologies, to include both DSSS (Direct Sequence Spread Spectrum) and FHSS (Frequency Hopping Spread Spectrum) are known in the art. Spread spectrum commonly employs Multi-Carrier Modulation (MCM) such as OFDM (Orthogonal Frequency Division Multiplexing). OFDM and other spread spectrum are commonly used in wireless communication systems, particularly in WLAN networks.

Bluetooth. Bluetooth is a wireless technology standard for exchanging data over short distances (using short-wavelength UHF radio waves in the ISM band from 2.4 to 2.485 GHz) from fixed and mobile devices, and building personal area networks (PANs). It can connect several devices, overcoming problems of synchronization. A Personal Area Network (PAN) may be according to, compatible with, or based on, Bluetooth™ or IEEE 802.15.1-2005 standard. A Bluetooth controlled electrical appliance is described in U.S. Patent Application No. 2014/0159877 to Huang entitled: “Bluetooth Controllable Electrical Appliance”, and an electric power supply is described in U.S. Patent Application No. 2014/0070613 to Garb et al. entitled: “Electric Power Supply and Related Methods”, which are both incorporated in their entirety for all purposes as if fully set forth herein. Any Personal Area Network (PAN) may be according to, compatible with, or based on, Bluetooth™ or IEEE 802.15.1-2005 standard. A Bluetooth controlled electrical appliance is described in U.S. Patent Application No. 2014/0159877 to Huang entitled: “Bluetooth Controllable Electrical Appliance”, and an electric power supply is described in U.S. Patent Application No. 2014/0070613 to Garb et al. entitled: “Electric Power Supply and Related Methods”, which are both incorporated in their entirety for all purposes as if fully set forth herein.

Bluetooth operates at frequencies between 2402 and 2480 MHz, or 2400 and 2483.5 MHz including guard bands 2 MHz wide at the bottom end and 3.5 MHz wide at the top. This is in the globally unlicensed (but not unregulated) Industrial, Scientific and Medical (ISM) 2.4 GHz short-range radio frequency band. Bluetooth uses a radio technology called frequency-hopping spread spectrum. Bluetooth divides transmitted data into packets, and transmits each packet on one of 79 designated Bluetooth channels. Each channel has a bandwidth of 1 MHz. It usually performs 800 hops per second, with Adaptive Frequency-Hopping (AFH) enabled. Bluetooth low energy uses 2 MHz spacing, which accommodates 40 channels. Bluetooth is a packet-based protocol with a master-slave structure. One master may communicate with up to seven slaves in a piconet. All devices share the master's clock. Packet exchange is based on the basic clock, defined by the master, which ticks at 312.5 is intervals. Two clock ticks make up a slot of 625 μs, and two slots make up a slot pair of 1250 μs. In the simple case of single-slot packets the master transmits in even slots and receives in odd slots. The slave, conversely, receives in even slots and transmits in odd slots. Packets may be 1, 3 or 5 slots long, but in all cases the master's transmission begins in even slots and the slave's in odd slots.

A master Bluetooth device can communicate with a maximum of seven devices in a piconet (an ad-hoc computer network using Bluetooth technology), though not all devices reach this maximum. The devices can switch roles, by agreement, and the slave can become the master (for example, a headset initiating a connection to a phone necessarily begins as master—as initiator of the connection—but may subsequently operate as slave). The Bluetooth Core Specification provides for the connection of two or more piconets to form a scatternet, in which certain devices simultaneously play the master role in one piconet and the slave role in another. At any given time, data can be transferred between the master and one other device (except for the little-used broadcast mode). The master chooses which slave device to address; typically, it switches rapidly from one device to another in a round-robin fashion. Since it is the master that chooses which slave to address, whereas a slave is supposed to listen in each receive slot, being a master is a lighter burden than being a slave. Being a master of seven slaves is possible; being a slave of more than one master is difficult.

Bluetooth Low Energy. Bluetooth low energy (Bluetooth LE, BLE, marketed as Bluetooth Smart) is a wireless personal area network technology designed and marketed by the Bluetooth Special Interest Group (SIG) aimed at novel applications in the healthcare, fitness, beacons, security, and home entertainment industries. Compared to Classic Bluetooth, Bluetooth Smart is intended to provide considerably reduced power consumption and cost while maintaining a similar communication range. Bluetooth low energy is described in a Bluetooth SIG published Dec. 2, 2014 standard Covered Core Package version: 4.2, entitled: “ Master Table of Contents & Compliance Requirements—Specification Volume 0”, and in an article published 2012 in Sensors [ISSN 1424-8220] by CarLes Gomez et al. [Sensors 2012, 12, 11734-11753; doi:10.3390/s120211734] entitled: “ Overview and Evaluation of Bluetooth Low Energy: An Emerging Low - Power Wireless Technology ”, which are both incorporated in their entirety for all purposes as if fully set forth herein.

Bluetooth Smart technology operates in the same spectrum range (the 2.400 GHz-2.4835 GHz ISM band) as Classic Bluetooth technology, but uses a different set of channels. Instead of the Classic Bluetooth 79 1-MHz channels, Bluetooth Smart has 40 2-MHz channels. Within a channel, data is transmitted using Gaussian frequency shift modulation, similar to Classic Bluetooth's Basic Rate scheme. The bit rate is 1 Mbit/s, and the maximum transmit power is 10 mW. Bluetooth Smart uses frequency hopping to counteract narrowband interference problems. Classic Bluetooth also uses frequency hopping but the details are different; as a result, while both FCC and ETSI classify Bluetooth technology as an FHSS scheme, Bluetooth Smart is classified as a system using digital modulation techniques or a direct-sequence spread spectrum. All Bluetooth Smart devices use the Generic Attribute Profile (GATT). The application programming interface offered by a Bluetooth Smart aware operating system will typically be based around GATT concepts.

Cellular. Cellular telephone network may be according to, compatible with, or may be based on, a Third Generation (3G) network that uses Universal Mobile Telecommunications System (UMTS), Wideband Code Division Multiple Access (W-CDMA) UMTS, High Speed Packet Access (HSPA), UMTS Time-Division Duplexing (TDD), CDMA2000 1×RTT, Evolution-Data Optimized (EV-DO), Global System for Mobile communications (GSM), or Enhanced Data rates for GSM Evolution (EDGE) EDGE-Evolution. Further, a cellular telephone network is a Fourth Generation (4G) network that uses Evolved High Speed Packet Access (HSPA+), Mobile Worldwide Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE), LTE-Advanced, Mobile Broadband Wireless Access (MBWA), or is based on IEEE 802.20-2008.

5G. 5G refers to the fifth generation technology standard for cellular networks, as the successor to the 4G networks which provide connectivity to most current cellphones. 5G networks are cellular networks, in which the service area is divided into small geographical areas called cells. All 5G wireless devices in a cell are connected to the Internet and telephone network by radio waves through a local antenna in the cell. The main advantage of the new networks is that they will have greater bandwidth, giving higher download speeds, eventually up to 10 gigabits per second (Gbit/s).

The increased speed is achieved partly by using higher-frequency radio waves than current cellular networks. However, higher-frequency radio waves have a shorter range than the frequencies used by previous cell phone towers, requiring smaller cells. So to ensure wide service, 5G networks operate on up to three frequency bands, low, medium, and high. A 5G network will be composed of networks of up to 3 different types of cells, each requiring different antennas, each type giving a different tradeoff of download speed vs. distance and service area. 5G cellphones and wireless devices will connect to the network through the highest speed antenna within range at their location:

Low-band 5G uses a similar frequency range to current 4G cellphones, 600-700 MHz, giving download speeds a little higher than 4G: 30-250 Megabits per Second (Mbit/s). Low-band cell towers will have a range and coverage area similar to current 4G towers. Mid-band 5G uses microwaves of 2.5-3.7 GHz, currently allowing speeds of 100-900 Mbit/s, with each cell tower providing service up to several miles in radius. High-band 5G currently uses frequencies of 25-39 GHz, near the bottom of the millimeter wave band, although higher frequencies may be used in the future. It often achieves download speeds of a gigabit per second (Gbit/s), comparable to cable internet. The industry consortium setting standards for 5G is the 3rd Generation Partnership Project (3GPP).

Random. Randomness is commonly implemented by using random numbers, defined as a sequence of numbers or symbols that lack any pattern and thus appear random, are often generated by a random number generator. Randomness for security is also described in IETF RFC 1750 “Randomness Recommendations for Security ” (December 2094), which is incorporated in its entirety for all purposes as if fully set forth herein. A random number generator (having either analog or digital output) can be hardware based, using a physical process such as thermal noise, shot noise, nuclear decaying radiation, photoelectric effect or other quantum phenomena. Alternatively, or in addition, the generation of the random numbers can be software based, using a processor executing an algorithm for generating pseudo-random numbers which approximates the properties of random numbers.

The term ‘random’ herein is intended to cover not only pure random, non-deterministically and non-predicted generated signals, but also pseudo-random, deterministic signals such as the output of a shift-register arrangement provided with a feedback circuit as used to generate pseudo-random binary signals or as scramblers, and chaotic signals, and where a randomness factor may be used.

A digital random signal generator (known as random number generator) wherein numbers in binary form replaces the analog voltage value output may be used for any randomness. One approach to random number generation is based on using linear feedback shift registers. An example of random number generators is disclosed in U.S. Pat. No. 7,124,157 to Ikake entitled: “Random Number Generator”, in U.S. Pat. No. 4,905,176 to Schulz entitled: “Random Number Generator Circuit”, in U.S. Pat. No. 4,853,884 to Brown et al. entitled: “Random Number Generator with Digital Feedback” and in U.S. Pat. No. 7,145,933 to Szajnowski entitled: “Method and Apparatus for generating Random signals”, which are incorporated in its entirety for all purposes as if fully set forth herein.

A digital random signal generator may be based on ‘True Random Number Generation IC RPG100/RPG100B’ available from FDK Corporation and described in the data sheet ‘Physical Random number generator RPG100.RPG100B’ REV. 08 publication number HM-RAE106-0812, which is incorporated in its entirety for all purposes as if fully set forth herein. The digital random signal generator can be hardware based, generating random numbers from a natural physical process or phenomenon, such as the thermal noise of semiconductor which has no periodicity. Typically, such hardware random number generators are based on microscopic phenomena such as thermal noise, shot noise, nuclear decaying radiation, photoelectric effect or other quantum phenomena, and typically contain a transducer to convert some aspect of the physical phenomenon to an electrical signal, an amplifier and other electronic to bring the output into a signal that can be converted into a digital representation by an analog to digital converter. In the case where digitized serial random number signals are generated, the output is converted to parallel, such as 8 bits data, with 256 values of random numbers (values from 0 to 255). Alternatively, a digital random signal generator may be software (or firmware) based, such as pseudo-random number generators. Such generators include a processor for executing software that includes an algorithm for generating numbers, which approximates the properties of random numbers. The random signal generator (either analog or digital) may output a signal having uniform distribution, in which there is a substantially or purely equal probability of a signal falling between two defined limits, having no appearance outside these limits. However, Gaussian and other distribution may be equally used.

Microphone. A transducer is a device for converting one form of energy into another. In an electroacoustic context, this means converting sound energy into electrical energy (or vice versa). Electroacoustic transducers include loudspeakers, microphones, hydrophones, and sonar projectors. These devices convert a sound pressure wave to or from an electric signal, and the most widely used transduction principles are electromagnetism, electrostatics and piezoelectricity. The transducers in most common loudspeakers (e.g. woofers and tweeters), are electromagnetic devices that generate waves using a suspended diaphragm driven by an electromagnetic voice coil, sending off pressure waves. Electret microphones and condenser microphones employ electrostatics—as the sound wave strikes the microphone's diaphragm, it moves and induces a voltage change. The ultrasonic systems used in medical ultrasonography employ piezoelectric transducers. These are made from special ceramics in which mechanical vibrations and electrical fields are interlinked through a property of the material itself.

In a common technique of acoustic measurement, acoustic signals are sampled in time, and then presented in more meaningful forms such as octave bands or time frequency plots. The entire spectrum can be divided into three sections: audio, ultrasonic, and infrasonic. The audio range falls between 20 Hz and 20,000 Hz, and is important because its frequencies can be detected by the human ear. This range has a number of applications, including speech communication and music. The ultrasonic range refers to the very high frequencies: 20,000 Hz and higher, and this range has shorter wavelengths which allow better resolution in imaging technologies. On the other end of the spectrum, the lowest frequencies are known as the infrasonic range, and these frequencies can be used to study geological phenomena such as earthquakes.

A microphone is an electroacoustic sensor that responds to sound waves (which are essentially vibrations transmitted through an elastic solid or a liquid or gas), and converts sound into electrical energy, usually by means of a ribbon or diaphragm set into motion by the sound waves. The sound may be audio or audible, having frequencies in the approximate range of 20 to 20,000 hertz, capable of being detected by human organs of hearing. Alternatively or in addition, the microphone may be used to sense inaudible frequencies, such as ultrasonic (a.k.a. ultrasound) acoustic frequencies that are above the range audible to the human ear, or above approximately 20,000 Hz. A microphone may be a condenser microphone (a.k.a. capacitor or electrostatic microphone) where the diaphragm acts as one plate of a two plates capacitor, and the vibrations changes the distance between plates, hence changing the capacitance. An electret microphone is a capacitor microphone based on a permanent charge of an electret or a polarized ferroelectric material. A dynamic microphone is based on electromagnetic induction, using a diaphragm attached to a small movable induction coil that is positioned in a magnetic field of a permanent magnet. The incident sound waves cause the diaphragm to vibrate, and the coil to move in the magnetic field, producing a current. Similarly, a ribbon microphone uses a thin, usually corrugated metal ribbon suspended in a magnetic field, and its vibration within the magnetic field generates the electrical signal. A loudspeaker is commonly constructed similar to a dynamic microphone, and thus may be used as a microphone as well. In a carbon microphone, the diaphragm vibrations apply varying pressure to a carbon, thus changing its electrical resistance. A piezoelectric microphone (a.k.a. crystal or piezo microphone) is based on the phenomenon of piezoelectricity in piezoelectric crystals such as potassium sodium tartrate. A microphone may be omnidirectional, unidirectional, bidirectional, or provide other directionality or polar patterns.

Noise-cancelling microphone. A noise-canceling microphone is a microphone that is designed to filter ambient noise from the desired sound, which is especially useful in noisy environments. The development is a special case of the differential microphone topology most commonly used to achieve directionality, and all such microphones have at least two ports through which sound enters; a front port normally oriented toward the desired sound and another port that's more distant. The microphone's diaphragm is placed between the two ports; sound arriving from an ambient sound field reaches both ports more or less equally. Sound that's much closer to the front port than to the rear will make more of a pressure gradient between the front and back of the diaphragm, causing it to move more. The microphone's proximity effect is adjusted so that flat frequency response is achieved for sound sources very close to the front of the mic—typically 1 to 3 cm. Sounds arriving from other angles are subject to steep midrange and bass rolloff.

Another technique uses two or more microphones and active or passive circuitry to reduce the noise. The primary microphone is closer to the desired source (like a person's mouth), while a second microphone receives ambient noise. In a noisy environment, both microphones receive noise at a similar level, but the primary mic receives the desired sounds more strongly. Thus if one signal is subtracted from the other (in the simplest sense, by connecting the microphones out of phase) much of the noise is canceled while the desired sound is retained. Other techniques may be used as well, such as using a directional primary mic, to maximize the difference between the two signals and make the cancellation easier to do. The internal electronic circuitry of an active noise-canceling mic attempts to subtract noise signal from the primary microphone. The circuit may employ passive or active noise canceling techniques to filter out the noise, producing an output signal that has a lower noise floor and a higher signal-to-noise ratio.

An improved noise canceling microphone including robust design features and advanced noise control and speech discrimination convergence characteristics is described in U.S. Pat. No. 7,248,708 to Vaudrey et al. entitled: “Noise canceling microphone”, which is incorporated in its entirety for all purposes as if fully set forth herein. Two adaptive controllers are used to ensure robust performance in quickly changing acoustic environments ensuring an acceptable minimum performance characteristic. Additionally, a new real-time spectral estimation procedure is applied to a noise canceling communications microphone platform that permits continued and optimal adaptation of non-voice bandwidth frequencies during speech transients.

Certain embodiments of a noise-cancelling microphone with acoustically tuned ports are disclosed in U.S. Pat. No. 7,162,041 to Haapapuro et al. entitled: “Noise canceling microphone with acoustically tuned ports”, which is incorporated in its entirety for all purposes as if fully set forth herein. The noise canceling microphone may comprise a housing, a transducer for converting received energy received into electrical signals, where the transducer is located in the housing, a front and rear sound pathways to a front and rear sound openings in the transducer, where the front and rear sound pathways may be located on opposite sides of the housing and may be displaced 180 degrees off a vertical axis. The noise canceling microphone may further comprising a boom for supporting the noise canceling microphone, where the boom may be deformed to place the noise canceling microphone near the mouth of the user. For example, the boom may be deformed to place the noise canceling microphone at least ten millimeters away from the edge of the mouth of the user.

Optical-based microphone. A fiber optic microphone converts acoustic waves into electrical signals by sensing changes in light intensity, instead of sensing changes in capacitance or magnetic fields as with conventional microphones. During operation, light from a laser source travels through an optical fiber to illuminate the surface of a reflective diaphragm. Sound vibrations of the diaphragm modulate the intensity of light reflecting off the diaphragm in a specific direction. The modulated light is then transmitted over a second optical fiber to a photo detector, which transforms the intensity-modulated light into analog or digital audio for transmission or recording. Fiber optic microphones possess high dynamic and frequency range, similar to the best high-fidelity conventional microphones. Fiber optic microphones do not react to or influence any electrical, magnetic, electrostatic or radioactive fields (this is called EMI/RFI immunity). The fiber optic microphone design is therefore ideal for use in areas where conventional microphones are ineffective or dangerous, such as inside industrial turbines or in Magnetic Resonance Imaging (MRI) equipment environments.

Fiber optic microphones are robust, resistant to environmental changes in heat and moisture, and can be produced for any directionality or impedance matching. The distance between the microphone's light source and its photo detector may be up to several kilometers without need for any preamplifier or other electrical device, making fiber optic microphones suitable for industrial and surveillance acoustic monitoring. Fiber optic microphones are used in very specific application areas such as for infrasound monitoring and noise-canceling. They have proven especially useful in medical applications, such as allowing radiologists, staff and patients within the powerful and noisy magnetic field to converse normally, inside the MRI suites as well as in remote control rooms. Other uses include industrial equipment monitoring and audio calibration and measurement, high-fidelity recording and law enforcement. An example of an optical microphone is IAS MO 2000 Set available from Sennheiser Electronic Corporation (Headquartered in Old Lyme, Conn. U.S.A.) and described in a Product Description entitled: “ MO 20000 Set and IAS MO 2000 Set ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

A head for an optical microphone/sensor is disclosed in U.S. Pat. No. 6,694,031 to Paritsky et al. entitled: “Optical Microphone/Sensors”, which is incorporated in its entirety for all purposes as if fully set forth herein. The head including first and second light guides; the first light guide being coupled at an input end to a source of light and having an output end portion for transmitting light onto a membrane; the second light guide having an input end portion for receiving light reflected from the membrane and an output end coupled to a photodetector; the output end and input end portions each having an upper face and side surfaces and being disposed in close proximity to each other and optically separated along adjacent surfaces; characterized in that in order to utilize maximum light energy transmitted through the light guides by the light source, reflected by the membrane and received by the photodetector, at least one of the faces or surfaces is configured to extend along one or more planes which differ from the plane including the axes of the transmission of the light energy emitted from the light source and received by the photodetector.

A method of making optical transducers A head for an optical microphone/sensor is disclosed in U.S. Pat. No. 6,822,750 to Paritsky et al. entitled: “Optical Transducers and Methods of Making Same”, which is incorporated in its entirety for all purposes as if fully set forth herein. The method involves producing an integrated structure including, in a rectangular matrix array, a plurality of discrete light sources, a plurality of discrete light detectors each laterally spaced from a light source, a light shield in the space between a light source and a light detector for shielding the light detector from direct exposure to the light source, and a transparent plastic potting material embedding the light sources, light detectors and light shield; and cutting the integrated structure, along lines of the matrix, into individual optical units, each including a light source, a light detector, a light shield therebetween all embedded in the transparent plastic potting material, and an optical window for outputting light from the light source and for transmitting to the light detector light reflected back from the light source. Also described are optical units of a structure facilitating mass production of such optical transducers and providing a sturdy construction permitting rough handling.

A small optical microphone/sensor for measuring distances to, and/or physical properties of, a reflective surface is disclosed in U.S. Pat. No. 6,462,808 to Paritsky et al. entitled: “Small Optical Microphone/Sensor”, which is incorporated in its entirety for all purposes as if fully set forth herein. The small optical microphone/sensor comprising a source of light coupled to a light waveguide for transmitting a light beam through the waveguide; the waveguide having at one of its ends a pointed face with an angle determined by Snell's Law of Refraction wherein al is the angle of travel of the light beam through the waveguide media; α2 is the angle of travel of the light beam in a second media when exiting from the pointed face, and n1 and n2 are the light indices of the light waveguide media and the second media; the reflective surface being disposed at an optimal distance from the pointed face as determined by the angle α2; the waveguide having, at its outer surface, at least adjacent to the pointed face, means for preventing light waves impinging on the surface from being reflected back into the waveguide, and a light detector arranged to receive the light reflected from the surface.

An optical microphone for detecting an acoustic wave propagating in an ambient fluid is disclosed in U.S. Pat. No. 8,813,571 to Iwamoto et al. entitled: “Optical Microphone”, which is incorporated in its entirety for all purposes as if fully set forth herein. The optical microphone including: a propagation medium section; a light source for emitting a light wave to be transmitted through a diffraction region in the propagation medium section; and a photoelectric conversion section for detecting the light wave having been transmitted through the propagation medium section. A first acoustic wave which is a portion of the acoustic wave and a second acoustic wave which is another portion thereof are allowed to propagate in the propagation medium section so as to simultaneously arrive at the diffraction region, and an interference component between a +1st order diffracted light wave and a −1st order diffracted light wave of the light wave generated based on a refractive index distribution of the propagation medium occurring in the diffraction region.

Sensors array. Multiple sensors may be used arranged as a sensor array (such as linear sensor array), for improving the sensitivity, accuracy, resolution, and other parameters of the sensed phenomenon. The sensor array may be directional, and better measure the parameters of the impinging signal to the array, such as the number, magnitudes, frequencies, Direction-Of-Arrival (DOA), distances, and speeds of the signals. The processing of the entire sensor array outputs, such as to obtain a single measurement or a single parameter, may be performed by a dedicated processor, which may be part of the sensor array assembly. The same component may serve both as a sensor and as actuator, such as during different times, and may be associated with the same or different phenomenon. A sensor operation may be based on an external or integral mechanism for generating a stimulus or an excitation to generate influence or create a phenomenon.

Microphone array. A microphone array is any number of microphones operating in tandem. There are many applications such as systems for extracting voice input from ambient noise (notably telephones, speech recognition systems, hearing aids), surround sound and related technologies, binaural recording, locating objects by sound: acoustic source localization, e.g., military use to locate the source(s) of artillery fire, aircraft location, and tracking, and high fidelity original recordings. Typically, an array is made up of omnidirectional microphones, directional microphones, or a mix of omnidirectional and directional microphones distributed about the perimeter of a space, linked to a computer that records and interprets the results into a coherent form. Arrays may also be formed using numbers of very closely spaced microphones. Given a fixed physical relationship in space between the different individual microphone transducer array elements, simultaneous DSP (digital signal processor) processing of the signals from each of the individual microphone array elements can create one or more “virtual” microphones. Different algorithms permit the creation of virtual microphones with extremely complex virtual polar patterns and even the possibility to steer the individual lobes of the virtual microphones patterns to home-in-on, or to reject, particular sources of sound.

In case the array consists of omnidirectional microphones they accept sound from all directions, so electrical signals of the microphones contain the information about the sounds coming from all directions. Joint processing of these sounds allow selecting the sound signal coming from the given direction. Hence, microphone array selects the sound coming from a given direction by processing multichannel signals. Using microphone arrays is described in an article by Rainer Zelinski (of the Deutsche Bundespost, Research Institute Berlin) published 1998 by IEEE (CH2561-9/88/0000-2578) entitled: “ A Microphone Array with Adaptive Post - Filtering for Noise Reduction in Reverberant Rooms ”, and in a presentation by Sven Fischer et al. presented 2-6 Dec. 1996 at the 3rd Joint Meeting of the Acoustical Society of America and the Acoustical Society of Japan entitled: “ Adaptive Microphone Arrays for Speech Enhancement in Coherent and Incoherent Noise Fields ”, which are both incorporated in their entirety for all purposes as if fully set forth herein.

Display. A display is used for presentation of visual data or information, commonly on a screen. A display is typically consists of an array of light emitters (typically in a matrix form), and commonly provides a visual depiction of a single, integrated, or organized set of information, such as text, graphics, image or video. A display may be a monochrome (a.k.a. black-and-white) type, which typically displays two colors, one for the background and one for the foreground. Old computer monitor displays commonly use black and white, green and black, or amber and black. A display may be a gray-scale type, which is capable of displaying different shades of gray, or may be a color type, capable of displaying multiple colors, anywhere from 16 to over many millions different colors, and may be based on Red, Green, and Blue (RGB) separate signals. A video display is designed for presenting video content. The screen is the actual location where the information is actually optically visualized by humans. The screen may be an integral part of the display.

Alternatively or in addition, the display may be an image or video projector, that projects an image (or a video consisting of moving images) onto a screen surface, which is a separate component and is not mechanically enclosed with the display housing. Most projectors create an image by shining a light through a small transparent image, but some newer types of projectors can project the image directly, by using lasers. A projector may be based on an Eidophor, Liquid Crystal on Silicon (LCoS or LCOS), or LCD, or may use Digital Light Processing (DLP™) technology, and may further be MEMS based. A virtual retinal display, or retinal projector, is a projector that projects an image directly on the retina instead of using an external projection screen.

Common display resolutions used today include SVGA (800×600 pixels), XGA (1024×768 pixels), 720p (1280×720 pixels), and 1080p (1920×1080 pixels). Standard-Definition (SD) standards, such as used in SD Television (SDTV), are referred to as 576i, derived from the European-developed PAL and SECAM systems with 576 interlaced lines of resolution; and 480i, based on the American National Television System Committee (ANTSC) NTSC system. High-Definition (HD) video refers to any video system of higher resolution than standard-definition (SD) video, and most commonly involves display resolutions of 1,280×720 pixels (720p) or 1,920×1,080 pixels (1080i/1080p). A display may be a 3D (3-Dimensions) display, which is the display device capable of conveying a stereoscopic perception of 3-D depth to the viewer. The basic technique is to present offset images that are displayed separately to the left and right eye. Both of these 2-D offset images are then combined in the brain to give the perception of 3-D depth. The display may present the information as scrolling, static, bold or flashing.

A display may be an analog display having an analog signal input. Analog displays are commonly using interfaces such as composite video such as NTSC, PAL or SECAM formats. Similarly, analog RGB, VGA (Video Graphics Array), SVGA (Super Video Graphics Array), SCART, S-video and other standard analog interfaces can be used. Alternatively or in addition, a display may be a digital display, having a digital input interface. Standard digital interfaces such as an IEEE1394 interface (a.k.a. FireWire™), may be used. Other digital interfaces that can be used are USB, SDI (Serial Digital Interface), HDMI (High-Definition Multimedia Interface), DVI (Digital Visual Interface), UDI (Unified Display Interface), DisplayPort, Digital Component Video and DVB (Digital Video Broadcast). In some cases, an adaptor is required in order to connect an analog display to the digital data. For example, the adaptor may convert between composite video (PAL, NTSC) or S-Video and DVI or HDTV signal. Various user controls can be available to allow the user to control and effect the display operations, such as an on/off switch, a reset button and others. Other exemplary controls involve display associated settings such as contrast, brightness and zoom.

A display may be a Cathode-Ray Tube (CRT) display, which is based on moving an electron beam back and forth across the back of the screen. Such a display commonly comprises a vacuum tube containing an electron gun (a source of electrons), and a fluorescent screen used to view images. It further has a means to accelerate and deflect the electron beam onto the fluorescent screen to create the images. Each time the beam makes a pass across the screen, it lights up phosphor dots on the inside of the glass tube, thereby illuminating the active portions of the screen. By drawing many such lines from the top to the bottom of the screen, it creates an entire image. A CRT display may be a shadow mask or an aperture grille type.

A display may be a Liquid Crystal Display (LCD) display, which utilize two sheets of polarizing material with a liquid crystal solution between them. An electric current passed through the liquid causes the crystals to align so that light cannot pass through them. Each crystal, therefore, is like a shutter, either allowing a backlit light to pass through or blocking the light. In monochrome LCD, images usually appear as blue or dark gray images on top of a grayish-white background. Color LCD displays commonly use passive matrix and Thin Film Transistor (TFT) (or active-matrix) for producing color. Recent passive-matrix displays are using new CSTN and DSTN technologies to produce sharp colors rivaling active-matrix displays.

Some LCD displays use Cold-Cathode Fluorescent Lamps (CCFLs) for backlight illumination. An LED-backlit LCD is a flat panel display that uses LED backlighting instead of the cold cathode fluorescent (CCFL) backlighting, allowing for a thinner panel, lower power consumption, better heat dissipation, a brighter display, and better contrast levels. Three forms of LED may be used: White edge-LEDs around the rim of the screen, using a special diffusion panel to spread the light evenly behind the screen (the most usual form currently), an array of LEDs arranged behind the screen whose brightness are not controlled individually, and a dynamic “local dimming” array of LEDs that are controlled individually or in clusters to achieve a modulated backlight light pattern. A Blue Phase Mode LCD is an LCD technology that uses highly twisted cholesteric phases in a blue phase, in order to improve the temporal response of liquid crystal displays (LCDs).

A Field Emission Display (FED) is a display technology that uses large-area field electron emission sources to provide the electrons that strike colored phosphor, to produce a color image as an electronic visual display. In a general sense, a FED consists of a matrix of cathode ray tubes, each tube producing a single sub-pixel, grouped in threes to form red-green-blue (RGB) pixels. FEDs combine the advantages of CRTs, namely their high contrast levels and very fast response times, with the packaging advantages of LCD and other flat panel technologies. They also offer the possibility of requiring less power, about half that of an LCD system. FED display operates like a conventional cathode ray tube (CRT) with an electron gun that uses high voltage (10 kV) to accelerate electrons which in turn excite the phosphors, but instead of a single electron gun, a FED display contains a grid of individual nanoscopic electron guns. A FED screen is constructed by laying down a series of metal stripes onto a glass plate to form a series of cathode lines.

A display may be an Organic Light-Emitting Diode (OLED) display, a display device that sandwiches carbon-based films between two charged electrodes, one a metallic cathode and one a transparent anode, usually being glass. The organic films consist of a hole-injection layer, a hole-transport layer, an emissive layer and an electron-transport layer. When voltage is applied to the OLED cell, the injected positive and negative charges recombine in the emissive layer and create electro luminescent light. Unlike LCDs, which require backlighting, OLED displays are emissive devices—they emit light rather than modulate transmitted or reflected light. There are two main families of OLEDs: those based on small molecules and those employing polymers. Adding mobile ions to an OLED creates a light-emitting electrochemical cell or LEC, which has a slightly different mode of operation. OLED displays can use either Passive-Matrix (PMOLED) or active-matrix addressing schemes. Active-Matrix OLEDs (AMOLED) require a thin-film transistor backplane to switch each individual pixel on or off, but allow for higher resolution and larger display sizes.

A display may be an Electroluminescent Displays (ELDs) type, which is a flat panel display created by sandwiching a layer of electroluminescent material such as GaAs between two layers of conductors. When current flows, the layer of material emits radiation in the form of visible light. Electroluminescence (EL) is an optical and electrical phenomenon where a material emits light in response to an electric current passed through it, or to a strong electric field.

A display may be based on an Electronic Paper Display (EPD) (a.k.a. e-paper and electronic ink) display technology which is designed to mimic the appearance of ordinary ink on paper. Unlike conventional backlit flat panel displays which emit light, electronic paper displays reflect light like ordinary paper. Many of the technologies can hold static text and images indefinitely without using electricity, while allowing images to be changed later. Flexible electronic paper uses plastic substrates and plastic electronics for the display backplane.

An EPD may be based on Gyricon technology, using polyethylene spheres between 75 and 106 micrometers across. Each sphere is a janus particle composed of negatively charged black plastic on one side and positively charged white plastic on the other (each bead is thus a dipole). The spheres are embedded in a transparent silicone sheet, with each sphere suspended in a bubble of oil so that they can rotate freely. The polarity of the voltage applied to each pair of electrodes then determines whether the white or black side is face-up, thus giving the pixel a white or black appearance. Alternatively or in addition, an EPD may be based on an electrophoretic display, where titanium dioxide (Titania) particles approximately one micrometer in diameter are dispersed in hydrocarbon oil. A dark-colored dye is also added to the oil, along with surfactants and charging agents that cause the particles to take on an electric charge. This mixture is placed between two parallel, conductive plates separated by a gap of 10 to 100 micrometers. When a voltage is applied across the two plates, the particles will migrate electrophoretically to the plate bearing the opposite charge from that on the particles.

Further, an EPD may be based on Electro-Wetting Display (EWD), which is based on controlling the shape of a confined water/oil interface by an applied voltage. With no voltage applied, the (colored) oil forms a flat film between the water and a hydrophobic (water-repellent) insulating coating of an electrode, resulting in a colored pixel. When a voltage is applied between the electrode and the water, it changes the interfacial tension between the water and the coating. As a result, the stacked state is no longer stable, causing the water to move the oil aside. Electrofluidic displays are a variation of an electrowetting display, involving the placing of aqueous pigment dispersion inside a tiny reservoir. Voltage is used to electromechanically pull the pigment out of the reservoir and spread it as a film directly behind the viewing substrate. As a result, the display takes on color and brightness similar to that of conventional pigments printed on paper. When voltage is removed liquid surface tension causes the pigment dispersion to rapidly recoil into the reservoir.

A display may be a Vacuum Fluorescent Display (VFD) that emits a very bright light with high contrast and can support display elements of various colors. VFDs can display seven-segment numerals, multi-segment alphanumeric characters or can be made in a dot-matrix to display different alphanumeric characters and symbols.

A display may be a laser video display or a laser video projector. A Laser display requires lasers in three distinct wavelengths—red, green, and blue. Frequency doubling can be used to provide the green wavelengths, and a small semiconductor laser such as Vertical-External-Cavity Surface-Emitting-Laser (VECSEL) or a Vertical-Cavity Surface-Emitting Laser (VCSEL) may be used. Several types of lasers can be used as the frequency doubled sources: fiber lasers, inter cavity doubled lasers, external cavity doubled lasers, eVCSELs, and OPSLs (Optically Pumped Semiconductor Lasers). Among the inter-cavity doubled lasers VCSELs have shown much promise and potential to be the basis for a mass-produced frequency doubled laser. A VECSEL is a vertical cavity, and is composed of two mirrors. On top of one of them is a diode as the active medium. These lasers combine high overall efficiency with good beam quality. The light from the high-power IR-laser diodes is converted into visible light by means of extra-cavity waveguided second harmonic generation. Laser-pulses with about 10 KHz repetition rate and various lengths are sent to a Digital Micromirror Device where each mirror directs the pulse either onto the screen or into the dump.

A display may be a segment display, such as a numerical or an alphanumerical display that can show only digits or alphanumeric characters, commonly composed of several segments that switch on and off to give the appearance of desired glyph, The segments are usually single LEDs or liquid crystals, and may further display visual display material beyond words and characters, such as arrows, symbols, ASCII and non-ASCII characters. Non-limiting examples are Seven-segment display (digits only), Fourteen-segment display, and Sixteen-segment display. A display may be a dot matrix display, used to display information on machines, clocks, railway departure indicators and many other devices requiring a simple display device of limited resolution. The display consists of a matrix of lights or mechanical indicators arranged in a rectangular configuration (other shapes are also possible, although not common) such that by switching on or off selected lights, text or graphics can be displayed. A dot matrix controller converts instructions from a processor into signals which turns on or off the lights in the matrix so that the required display is produced.

Sounder. A sounder converts electrical energy to sound waves transmitted through the air, an elastic solid material, or a liquid, usually by means of a vibrating or moving ribbon or diaphragm. The sound may be audio or audible, having frequencies in the approximate range of 20 to 20,000 hertz, capable of being detected by human organs of hearing. Alternatively or in addition, the sounder may be used to emit inaudible frequencies, such as ultrasonic (a.k.a. ultrasound) acoustic frequencies that are above the range audible to the human ear, or above approximately 20,000 Hz. A sounder may be omnidirectional, unidirectional, bidirectional, or provide other directionality or polar patterns.

A loudspeaker (a.k.a. speaker) is a sounder that produces sound in response to an electrical audio signal input, typically audible sound. The most common form of loudspeaker is the electromagnetic (or dynamic) type, uses a paper cone supporting a moving voice coil electromagnet acting on a permanent magnet. Where accurate reproduction of sound is required, multiple loudspeakers may be used, each reproducing a part of the audible frequency range. A loudspeaker is commonly optimized for middle frequencies; tweeters for high frequencies; and sometimes supertweeter is used which is optimized for the highest audible frequencies.

A loudspeaker may be a piezo (or piezoelectric) speaker contains a piezoelectric crystal coupled to a mechanical diaphragm and is based on the piezoelectric effect. An audio signal is applied to the crystal, which responds by flexing in proportion to the voltage applied across the crystal surfaces, thus converting electrical energy into mechanical. Piezoelectric speakers are frequently used as beepers in watches and other electronic devices, and are sometimes used as tweeters in less-expensive speaker systems, such as computer speakers and portable radios. A loudspeaker may be a magnetostrictive transducers, based on magnetostriction, have been predominantly used as sonar ultrasonic sound wave radiators, but their usage has spread also to audio speaker systems.

A loudspeaker may be an Electro-Static Loudspeaker (ESL), in which sound is generated by the force exerted on a membrane suspended in an electrostatic field. Such speakers use a thin flat diaphragm usually consisting of a plastic sheet coated with a conductive material such as graphite sandwiched between two electrically conductive grids, with a small air gap between the diaphragm and grids. The diaphragm is usually made from a polyester film (thickness 2-20 μm) with exceptional mechanical properties, such as PET film. By means of the conductive coating and an external high voltage supply the diaphragm is held at a DC potential of several kilovolts with respect to the grids. The grids are driven by the audio signal; and the front and rear grids are driven in antiphase. As a result, a uniform electrostatic field proportional to the audio signal is produced between both grids. This causes a force to be exerted on the charged diaphragm, and its resulting movement drives the air on either side of it.

A loudspeaker may be a magnetic loudspeaker, and may be a ribbon or planar type, is based on a magnetic field. A ribbon speaker consists of a thin metal-film ribbon suspended in a magnetic field. The electrical signal is applied to the ribbon, which moves with it to create the sound. Planar magnetic speakers are speakers with roughly rectangular flat surfaces that radiate in a bipolar (i.e., front and back) manner, and may be having printed or embedded conductors on a flat diaphragm. Planar magnetic speakers consist of a flexible membrane with a voice coil printed or mounted on it. The current flowing through the coil interacts with the magnetic field of carefully placed magnets on either side of the diaphragm, causing the membrane to vibrate more uniformly and without much bending or wrinkling. A loudspeaker may be a bending wave loudspeaker, which uses a diaphragm that is intentionally flexible.

A sounder may an electromechanical type, such as an electric bell, which may be based on an electromagnet, causing a metal ball to clap on cup or half-sphere bell. A sounder may be a buzzer (or beeper), a chime, a whistle or a ringer. Buzzers may be either electromechanical or ceramic-based piezoelectric sounders which make a high-pitch noise, and may be used for alerting. The sounder may emit a single or multiple tones, and can be in continuous or intermittent operation.

In one example, the sounder is used to play a stored digital audio. The digital audio content can be stored in the sounder, the actuator unit, the router, the control server, or any combination thereof. Further, few files may be stored (e.g., representing different announcements or songs), selected by the control logic. Alternatively or in addition, the digital audio data may be received by the sounder, the actuator unit, the router, the control server, or any combination thereof, from external sources via the above networks. Furthermore, the source of the digital audio may a microphone serving as a sensor, either after processing, storing, delaying, or any other manipulation, or as originally received resulting ‘doorphone’ or ‘intercom’ functionality between a microphone and a sounder in the building.

In another example, the sounder simulates the voice of a human being or generates music, typically by using an electronic circuit having a memory for storing the sounds (e.g., music, song, voice message, etc.), a digital to analog converter to reconstruct the electrical representation of the sound, and a driver for driving a loudspeaker, which is an electro-acoustic transducer that converts an electrical signal to sound. An example of a greeting card providing music and mechanical movement is disclosed in U.S. Patent Application No. 2007/0256337 to Segan entitled: “User Interactive Greeting Card”, which is incorporated in its entirety for all purposes as if fully set forth herein.

It is noted that the expression “sound information” or “sound” as used herein may refer to acoustic wave energy produced when playing an instrument and/or singing. Some systems and methods may also operate in static mode, in which no BGM music is played, and the user plays the notes at his own pace. In some embodiments, a cursor may be displayed, which advances in accordance with the user's playing or with the expected pace.

In one example, the system is used for sound or music generation. For example, the sound produced can emulate the sounds of a conventional acoustical music instrument, such as a piano, tuba, harp, violin, flute, guitar and so forth. In one example, the sounder is an audible signaling device, emitting audible sounds that can be heard (having frequency components in the 20-20,000 Hz band). In one example the sound generated is music or song. The elements of the music such as pitch (which governs melody and harmony), rhythm (and its associated concepts tempo, meter, and articulation), dynamics, and the sonic qualities of timbre and texture, may be associated with the shape theme. For example, if a musical instrument shown in the picture, the music generated by that instrument will be played, e.g., drumming sound of drums and playing of a flute or guitar. In one example, a talking human voice is played by the sounder. The sound may be a syllable, a word, a phrase, a sentence, a short story or a long story, and can be based on speech synthesis or prerecorded. Male or female voice can be used, further being young or old.

Some examples of toys that include generation of an audio signal such as music are disclosed in U.S. Pat. No. 4,496,149 to Schwartzberg entitled: “Game Apparatus Utilizing Controllable Audio Signals”, in U.S. Pat. No. 4,516,260 to Breedlove et al. entitled: “Electronic Learning Aid or Game having Synthesized Speech”, in U.S. Pat. No. 7,414,186 to Scarpa et al. entitled: “System and Method for Teaching Musical Notes”, in U.S. Pat. No. 4,968,255 to Lee et al., entitled: “Electronic Instructional Apparatus”, in U.S. Pat. No. 4,248,123 to Bunger et al., entitled: “Electronic Piano” and in U.S. Pat. No. 4,796,891 to Milner entitled: “Musical Puzzle Using Sliding Tiles”, and toys with means for synthesizing human voice are disclosed in U.S. Pat. No. 6,527,611 to Cummings entitled: “Place and Find Toy”, and in U.S. Pat. No. 4,840,602 to Rose entitled: “Talking Doll Responsive to External Signal”, which are all incorporated in their entirety for all purposes as if fully set forth herein. A music toy kit combining music toy instrument with a set of construction toy blocks is disclosed in U.S. Pat. No. 6,132,281 to Klitsner et al. entitled: “Music Toy Kit” and in U.S. Pat. No. 5,349,129 to Wisniewski et al. entitled: “Electronic Sound Generating Toy”, which are incorporated in their entirety for all purposes as if fully set forth herein.

A speech synthesizer used to produce natural and intelligible artificial human speech may be implemented in hardware, in software, or combination thereof. A speech synthesizer may be Text-To-Speech (TTS) based, that converts normal language text to speech, or alternatively (or in addition) may be based on rendering symbolic linguistic representation like phonetic transcription. A TTS typically involves two steps, the front-end where the raw input text is pre-processed to fully write-out words replacing numbers and abbreviations, followed by assigning phonetic transcriptions to each word (text-to-phoneme), and the back-end (or synthesizer) where the symbolic linguistic representation is converted to output sound.

The generating of synthetic speech waveform typically uses a concatenative or formant synthesis. The concatenative synthesis commonly produces the most natural-sounding synthesized speech, and is based on the concatenation (or stringing together) of segments of recorded speech. There are three main types of concatenative synthesis: Unit selection, diphone synthesis, and domain-specific synthesis. Unit selection synthesis is based on large databases of recorded speech including individual phones, diphones, half-phones, syllables, morphemes, words, phrases, and sentences, indexed based on the segmentation and acoustic parameters like the fundamental frequency (pitch), duration, position in the syllable, and neighboring phones. At run time, the desired target utterance is created by determining (typically using a specially weighted decision tree) the best chain of candidate units from the database (unit selection). Diphone synthesis uses a minimal speech database containing all the diphones (sound-to-sound transitions) occurring in a language, and at runtime, the target prosody of a sentence is superimposed on these minimal units by means of digital signal processing techniques such as linear predictive coding. Domain-specific synthesis is used where the output is limited to a particular domain, using concatenates prerecorded words and phrases to create complete utterances. In formant synthesis the synthesized speech output is created using additive synthesis and an acoustic model (physical modeling synthesis), rather than on using human speech samples. Parameters such as fundamental frequency, voicing, and noise levels are varied over time to create a waveform of artificial speech. The synthesis may further be based on articulatory synthesis where computational techniques for synthesizing speech are based on models of the human vocal tract and the articulation processes occurring there, or may be HMM-based synthesis which is based on hidden Markov models, where the frequency spectrum (vocal tract), fundamental frequency (vocal source), and duration (prosody) of speech are modeled simultaneously by HMMs and generated based on the maximum likelihood criterion. The speech synthesizer may further be based on the book entitled: “Development in Speech Synthesis”, by Mark Tatham and Katherine Morton, published 2005 by John Wiley & Sons Ltd., ISBN: 0-470-85538-X, and on the book entitled: “ Speech Synthesis and Recognition ” by John Holmes and Wendy Holmes, 2 nd Edition, published 2001 ISBN: 0-7484-0856-8, which are both incorporated in their entirety for all purposes as if fully set forth herein.

Haptic. Haptic technology, also known as kinaesthetic communication or 3D touch, refers to any technology that can create an experience of touch by applying forces, vibrations, or motions to the user. These technologies can be used to create virtual objects in a computer simulation, to control virtual objects, and to enhance remote control of machines and devices. Haptic devices may incorporate tactile sensors that measure forces exerted by the user on the interface, and simple haptic devices are common in the form of game controllers, joysticks, and steering wheels. Haptic technology facilitates investigation of how the human sense of touch works by allowing the creation of controlled haptic virtual objects. In general, three sensory systems related to sense of touch in humans are distinguished: cutaneous, kinaesthetic, and haptic.

The majority of electronics offering haptic feedback use vibrations, and most use a type of Eccentric Rotating Mass (ERM) actuator, consisting of an unbalanced weight attached to a motor shaft. As the shaft rotates, the spinning of this irregular mass causes the actuator and the attached device to shake. Some devices accomplish their vibrations with a Linear Resonant Actuator (LRA), which moves a mass in a reciprocal manner by means of a magnetic voice coil, similar to how AC electrical signals are translated into motion in the cone of a loudspeaker. LRAs are capable of quicker response times than ERMs, and thus can transmit more accurate haptic imagery. Piezoelectric actuators are also employed to produce vibrations, and offer even more precise motion than LRAs, with less noise and in a smaller platform, but require higher voltages than do ERMs and LRAs.

Some devices use motors to manipulate the movement of an item held by the user. A common use is in automobile driving video games and simulators, which turn the steering wheel to simulate forces experienced when cornering a real vehicle. Air vortex rings are donut-shaped air pockets made up of concentrated gusts of air. Focused air vortices can have the force to blow out a candle or disturb papers from a few yards away. Focused ultrasound beams can be used to create a localized sense of pressure on a finger without touching any physical object. The focal point that creates the sensation of pressure is generated by individually controlling the phase and intensity of each transducer in an array of ultrasound transducers. These beams can also be used to deliver sensations of vibration, and to give users the ability to feel virtual 3D objects.

Haptic rendering is described in a paper entitled: “ Introduction to Haptic Rendering ” by Miguel A. Otaduy and Ming C. Lin, published July 2005 [DOI: 10.1145/1198555.1198603], which is incorporated in its entirety for all purposes as if fully set forth herein. The term haptic the adjective used to describe something relating to or based on the sense of touch. Haptic is to touching as visual is to seeing and as auditory is to hearing. Typically, touch is one of the main avenues of sensation, and it can be divided into cutaneous, kinesthetic, and haptic systems, based on the underlying neural inputs. The cutaneous system employs receptors embedded in the skin, while the kinesthetic system employs receptors located in muscles, tendons, and joints. The haptic sensory system employs both cutaneous and kinesthetic receptors, but it differs in the sense that it is associated with an active procedure. Touch becomes active when the sensory inputs are combined with controlled body motion. For example, cutaneous touch becomes active when we explore a surface or grasp an object, while kinesthetic touch becomes active when we manipulate an object and touch other objects with it. Haptic rendering is defined as the process of computing and generating forces in response to user interactions with virtual objects. Several haptic rendering algorithms consider the paradigm of touching virtual objects with a single contact point. Rendering algorithms that follow this description are called 3-DoF haptic rendering algorithms, because a point in 3D has only three DoFs. Other haptic rendering algorithms deal with the problem of rendering the forces and torques arising from the interaction of two virtual objects. This problem is called 6-DoF haptic rendering, because the grasped object has six DoFs (position and orientation in 3D), and the haptic feedback comprises 3D force and torque. When we eat with a fork, write with a pen, or open a lock with a key, we are moving an object in 3D, and we feel the interaction with other objects. This is, in essence, 6-DoF object manipulation with force-and-torque feedback.

Characteristics of kinesthetic systems including their main function, historical evolution, commonly used technologies for construction and specific cases, and an analogical development of tactile systems, are described in an article entitled: “ HAPTIC INTERFACES: KINESTHETIC VS. TACTILE SYSTEMS ” by Zasúlich Pérez Ariza and Mauricio Santis-Chaves published 2016 [DOI: https://doi.org/10.24050/reia.v13i26.1065] in EIA, ISSN 1794-1237/Year XIII/Volume 13/Issue N.26/July-December 2016, which is incorporated in its entirety for all purposes as if fully set forth herein.

FFT. A Fast Fourier Transform (FFT) algorithm computes the discrete Fourier transform (DFT) of a sequence, or the inverse. Fourier analysis converts a signal from its original domain (often time or space) to a representation in the frequency domain and vice versa. An FFT rapidly computes such transformations by factorizing the DFT matrix into a product of sparse (mostly zero) factors. The DFT is obtained by decomposing a sequence of values into components of different frequencies. This operation is useful in many fields (see discrete Fourier transform for properties and applications of the transform) but computing it directly from the definition is often too slow to be practical. An FFT is a way to compute the same result more quickly: computing the DFT of N points in the naive way, using the definition, takes O(N2) arithmetical operations, while an FFT can compute the same DFT in only O(N log N) operations. The difference in speed can be enormous, especially for long data sets where N may be in the thousands or millions. In practice, the computation time can be reduced by several orders of magnitude in such cases, and the improvement is roughly proportional to N/log N. This huge improvement made the calculation of the DFT practical; FFTs are of great importance to a wide variety of applications, from digital signal processing and solving partial differential equations to algorithms for quick multiplication of large integers.

By far the most commonly used FFT is the Cooley-Tukey algorithm. This is a divide and conquer algorithm that recursively breaks down a DFT of any composite size N=N1N2 into many smaller DFTs of sizes N1 and N2, along with O(N) multiplications by complex roots of unity traditionally called twiddle factors. The best known use of the Cooley-Tukey algorithm is to divide the transform into two pieces of size N/2 at each step, and is therefore limited to power-of-two sizes, but any factorization can be used in general (as was known to both Gauss and Cooley/Tukey). These are called the radix-2 and mixed-radix cases, respectively (and other variants such as the split-radix FFT have their own names as well). Although the basic idea is recursive, most traditional implementations rearrange the algorithm to avoid explicit recursion. In addition, because the Cooley-Tukey algorithm breaks the DFT into smaller DFTs, it can be combined arbitrarily with any other algorithm for the DFT, such as those described below. FFT is described in an article by Paul Heckbert dated February 1995 (Revised 27 Jan. 1998) [Notes 3, Computer Graphics 2, 15-463] entitled: “ Fourier Transforms and the Fast Fourier Transform ( FFT ) Algorithm ”, and in Freescale Semiconductor, Inc. Application Note, Document Number AN4255 Rev. 4, July 2015, entitled: “ FFT - Based Algorithm for Metering Applications ”, which are both incorporated in their entirety for all purposes as if fully set forth herein.

Wearable. As used herein, the term “wearable device” (or “wearable”) includes a body-borne device (or item) designed or intended to be worn by a human. Such devices are typically comfortably worn on, and are carried or transported by, the human body, and are commonly used to create constant, convenient, seamless, portable, and mostly hands-free access to electronics and computers. The wearable devices may be in direct contact with the human body (such as by touching, or attaching to, the body skin), or may be releasably attachable to clothes or other items intended or designed to be worn on the human body. In general, the goal of wearable technologies is to smoothly incorporate functional, portable electronics and computers into individuals' daily lives. Wearable devices may be releasably attached to the human body using attaching means such as straps, buckles, belts, or clasps. Alternatively or in addition, wearable devices may be shaped, structured, or having a form factor to be body releasably mountable or attachable, such as using eye-glass frames or headphones. Further, wearable devices may be worn under, with, or on top of, clothing.

Wearable devices may interact as sensors or actuators with an organ or part of the human body, such as a head mounted wearable device may include a screen suspended in front of a user's eye, without providing any aid to the user's vision. Examples of wearable devices include watches, glasses, contact lenses, pedometers, chest straps, wrist-bands, head bands, arm bands, belt, head wear, hats, glasses, watches, sneakers, clothing, pads, e-textiles and smart fabrics, headbands, beanies, and caps, as well as jewelry such as rings, bracelets, and hearing aid-like devices that are designed to look like earrings. A wearable device may be structured, designed, or have a form factor that is identical to, substantially similar to, or is at least in part substitute to, a traditional wearable item.

A wearable device may be a headwear that may be structured, designed, or have a form factor that is identical to, substantially similar to, or is at least in part substitute to, any headwear item. The headwear may be attached to, or be in contact with, a head part, such as a face, nose, right nostril, left nostril, right cheek, left cheek, right eye, left eye, right ear, or left ear, nose, mouth, lip, forehead, or chin. A wearable device may be structured, designed, or have a form factor that is identical to, substantially similar to, or is at least in part substitute to, a bonnet, a cap, a crown, a fillet, a hair cover, a hat, a helmet, a hood, a mask, a turban, a veil, or a wig.

A headwear device may be an eyewear that may be structured, designed, or have a form factor that is identical to, substantially similar to, or is at least in part substitute to, any eyewear item, such as glasses, sunglasses, a contact lens, a blindfold, or a goggle. A headwear device may be an earpiece that may be structured, designed, or have a form factor that is identical to, substantially similar to, or is at least in part substitute to, any earpiece item, such as a hearing aid, a headphone, a headset, or an earplug.

A wearable device may be releasably or permanently attach to, or be part of, a clothing article such as a tie, sweater, jacket, or hat. The attachment may use taping, gluing, pinning, enclosing, encapsulating, or any other method of attachment or integration known in the art. Furthermore, in some embodiments, there may be an attachment element such as a pin or a latch and hook system, of portion thereof (with the complementary element on the item to which it is to be affixed) or clip. In a non-limiting example, the attachment element has a clip-like design to allow attachment to pockets, belts, watches, bracelets, broaches, rings, shoes, hats, bike handles, necklaces, ties, spectacles, collars, socks, bags, purses, wallets, or cords.

A wearable device may be releasably or permanently attach to, or be part of, a top underwear such as a bra, camisole, or undershirt, a bottom underwear such as a diaper, panties, plastic pants, slip, thong, underpants, boxer briefs, boxer shorts, or briefs, or a full-body underwear such as bodysuit, long underwear, playsuit, or teddy. Similarly, a wearable device may be releasably or permanently attach to, or be part of, a headwear such as a Baseball cap, Beret, Cap, Fedora, hat, helmet, hood, knit cap, toque, turban, or veil. Similarly, a wearable device may be releasably or permanently attach to, or be part of, a footwear such as an athletic shoe, boot, court shoe, dress shoe, flip-flops, hosiery, sandal, shoe, spats, slipper, sock, or stocking. Further, a wearable device may be releasably or permanently attach to, or be part of, an accessory such as a bandana, belt, bow tie, coin purse, cufflink, cummerbund, gaiters, glasses, gloves, headband, handbag, handkerchief, jewellery, muff, necktie, pocket protector, pocketwatch, sash, scarf, sunglasses, suspenders, umbrella, wallet, or wristwatch.

A wearable device may be releasably or permanently attach to, or be part of, an outwear such as an apron, blazer, British warm, cagoule, cape, chesterfield, coat, covert coat, cut-off, duffle coat, flight jacket, gilet, goggle jacket, guards coat, Harrington jacket, hoodie, jacket, leather jacket, mess jacket, opera coat, overcoat, parka, paletot, pea coat, poncho, raincoat, robe, safari jacket, shawl, shrug, ski suit, sleeved blanket, smoking jacket, sport coat, trench coat, ulster coat, waistcoat, or windbreaker. Similarly, a wearable device may be releasably or permanently attach to, or be part of, a suit (or uniform) such as an academic dress, ball dress, black tie, boilersuit, cleanroom suit, clerical clothing, court dress, gymslip, jumpsuit, kasaya, lab coat, military uniform, morning dress, onesie, pantsuit, red sea rig, romper suit, school uniform, scrubs, stroller, tuxedo, or white tie. Further, a wearable device may be releasably or permanently attach to, or be part of, a dress such as a ball gown, bouffant gown, coatdress, cocktail dress, debutante dress, formal wear, frock, evening gown, gown, house dress, jumper, little black dress, princess line, sheath dress, shirtdress, slip dress, strapless dress, sundress, wedding dress, or wrap dress. Furthermore, a wearable device may be releasably or permanently attach to, or be part of, a skirt such as an A-line skirt, ballerina skirt, denim skirt, men's skirts, miniskirt, pencil skirt, prairie skirt, rah-rah skirt, sarong, Skort, tutu, or wrap. In one example, a wearable device may be releasably or permanently attach to, or be part of, a trousers (or shorts) such as bell-bottoms, bermuda shorts, bondage pants, capri pants, cargo pants, chaps, cycling shorts, dress pants, high water pants, lowrise pants, Jeans, jodhpurs, leggings, overall, Palazzo pants, parachute pants, pedal pushers, phat pants, shorts, slim-fit pants, sweatpants, windpants, or yoga pants. In one example, a wearable device may be releasably or permanently attach to, or be part of, a top such as a blouse, crop top, dress shirt, guayabera, guernsey, halterneck, henley shirt, hoodie, jersey, polo shirt, shirt, sleeveless shirt, sweater, sweater vest, t-shirt, tube top, turtleneck, or twinset.

A wearable device may be structured, designed, or have a form factor that is identical to, substantially similar to, or is at least in part substitute to, a fashion accessory. These accessories may be purely decorative, or have a utility beyond aesthetics. Examples of these accessories include, but are not limited to, rings, bracelets, necklaces, watches, watch bands, purses, wallets, earrings, body rings, headbands, glasses, belts, ties, tie bars, tie tacks, wallets, shoes, pendants, charms and bobbles. For example, wearable devices may also be incorporated into pockets, steering wheels, keyboards, pens, and bicycle handles.

In one example, the wearable device may be shaped as, or integrated with, a device that includes an annular member defining an aperture therethrough that is sized for receipt therein of a human body part. The body part may be part of a human hand such as upper arm, elbow, forearm, wrist (such as a wrist-band), or a finger (such as a ring). Alternatively or in addition, the body part may be part of a human head or neck, such as a forehead, ear, skull, or face. Alternatively or in addition, the body part may be part of a human thorax or abdomen, such as waist or hip. Alternatively or in addition, the body part may be part of a human leg or foot, such as thigh, calf, ankle, instep, knee, or toe.

In one example, the wearable device may be shaped as, or integrated with, a ring. The ring may comprise, consist essentially of or consist of a shank, which is the location that provides an opening for a finger, and a head, which comprises, consists essentially or consists of ornamental features of the ring and in some embodiments houses the signaling assembly of the present device. The head may be of any shape, e.g., a regular sphere, truncated sphere, cube, rectangular prism, cylinder, triangular prism, cone, pyramid, barrel, truncated cone, domed cylinder, truncated cylinder, ellipsoid, regular polygon prism or truncated three-dimensional polygon of e.g., 4-16 sides, such as a truncated pyramid (trapezoid), or combination thereof or it may be an irregular shape. Further, the head may comprise an upper face that contains and is configured to show one or more jewels and/or ornamental designs.

A mobile communication device configured to be worn on an index finger of a user's hand is described in U.S. Patent Application Publication No. 2015/0373443 to Carroll entitled: “Finger-wearable mobile communication device”, which is incorporated in its entirety for all purposes as if fully set forth herein. The device includes a case, a microphone, a switch, and a power source. The microphone and the switch are strategically located along a shape of the case so that as worn on the user's index finger and when the switch is activated by the thumb of the user's hand, the hand naturally cups about the microphone to form a barrier to ambient noise. Further, the microphone can readily be located near a corner of the user's mouth for optimal speech-receiving conditions and to provide more private audio input.

A user controls an external electronic device with a finger-ring-mounted touchscreen is described in U.S. Patent Application Publication No. 2015/0277559 to Vescovi et al. entitled: “Devices and Methods for a Ring Computing Device”, which is incorporated in its entirety for all purposes as if fully set forth herein. The device includes a computer processor, wireless transceiver, and rechargeable power source; the ring is worn on a first finger receives an input from a second finger, selects one of a plurality of touch events associated with the input, and wirelessly transmits a command associated with the touch event to the external electronic device.

A mobile communication device that comprises a fashion accessory and a signaling assembly is described in U.S. Patent Application Publication No. 2015/0349556 to Mercando et al. entitled: “Mobile Communication Devices”, which is incorporated in its entirety for all purposes as if fully set forth herein. The signaling assembly may be configured to provide sensory stimuli such as a flashing LED light and a vibration. These stimuli may vary depending on the signal received from a remote communication device or from gestures made by a user or from information stored in the mobile communication device.

A wearable fitness-monitoring device is described in U.S. Pat. No. 8,948,832 to Hong et al. entitled: “Wearable Heart Rate Monitor”, which is incorporated in its entirety for all purposes as if fully set forth herein. The device including a motion sensor and a photoplethysmographic (PPG) sensor. The PPG sensor includes (i) a periodic light source, (ii) a photo detector, and (iii) circuitry determining a user's heart rate from an output of the photo detector. Some embodiments provide methods for operating a heart rate monitor of a wearable fitness-monitoring device to measure one or more characteristics of a heartbeat waveform. Some embodiments provide methods for operating the wearable fitness monitoring device in a low power state when the device determines that the device is not worn by a user. Some embodiments provide methods for operating the wearable fitness-monitoring device in a normal power state when the device determines that the device is worn by a user.

A wearable device and method for processing mages to prolong battery life are described in U.S. Pat. No. 8,957,988 to Wexler et al. entitled: “Apparatus for processing images to prolong battery life”, which is incorporated in its entirety for all purposes as if fully set forth herein. In one implementation, a wearable apparatus may include a wearable image sensor configured to capture a plurality of images from an environment of a user. The wearable apparatus may also include at least one processing device configured to, in a first processing-mode, process representations of the plurality of images to determine a value of at least one capturing parameter for use in capturing at least one subsequent image, and in a second processing-mode, process the representations of the plurality of images to extract information. In addition, the at least one processing device may operate in the first processing-mode when the wearable apparatus is powered by a mobile power source included in the wearable apparatus and may operate in the second processing-mode when the wearable apparatus is powered by an external power source.

A wearable device may be used for notifying a person, such as by using tactile, visual, or audible stimulus, as described for example in U.S. Patent Application No. 2015/0341901 to RYU et al. entitled: “Method and apparatus for providing notification”, which is incorporated in its entirety for all purposes as if fully set forth herein, describing an electronic device that includes: a transceiver configured to communicate with at least one wearable device and receive, from the at least one wearable device, status information indicating whether the at least one wearable device is currently being worn; and a processor configured to determine whether to send a notification request to the at least one wearable device based on the status information received by the transceiver.

A communication device, system and method are described for example in U.S. Patent Application No. 2007/0052672 to Ritter et al. entitled: “Communication device, system and method”, which is incorporated in its entirety for all purposes as if fully set forth herein. It is discloses comprising a Virtual Retinal Display (VRD) in form of glasses (1), at least one haptic sensor (12) mounted on the frame of said glasses or connected by a short range communication interface (13) to said glasses (1), wherein it is possible to navigate by means of a cursor through an image displayed by the Virtual Retinal Display (VRD) with the at least one haptic sensor (12). A central control unit controls (11) the Virtual Retinal Display (VRD) and the at least one haptic sensor (12). When the Virtual Retinal Display (VRD) is connected to an external device (2, 9) by a short range communication interface (13), the user can navigate through the content of the external device (2, 9) by easy use of the haptic sensor (12).

Wearable communication devices, e.g. implemented in a watch, using short range communication to a cell phone, and facilitating natural and intuitive user interface with low-power implementation are described for example in U.S. Patent Application No. 2014/0045547 to Singamsetty et al. entitled: “Wearable Communication Device and User Interface”, which is incorporated in its entirety for all purposes as if fully set forth herein. The devices allow a user to easily access all features of the phone, all while a phone is nearby but not visible. Notification is performed with vibration, an LED light and OLED text display of incoming calls, texts, and calendar events. It allows communicating hands-free. This allows using the communication device as “remote control” for home devices, etc. via voice and buttons. The device comprises interfaces motion sensors such as accelerometers, magnetometer and gyroscope, infrared proximity sensors, vibrator motor, and/or voice recognition. Low power consumption is achieved by dynamical configuration of sensor parameters to support only the necessary sensor functions at any given state of the device.

A wearable electronic device that is configured to control and command a variety of wireless devices within its proximity is described in U.S. Pat. No. 7,605,714 to Thompson et al. entitled: “System and method for command and control of wireless devices using a wearable device”, which is incorporated in its entirety for all purposes as if fully set forth herein. The wearable device dynamically generates a user interface corresponding to the services of a particular wireless device. Through the user interface, the wireless device surface content to a user and allows a user select interactions with the wireless devices using the wearable device.

An apparatus and method for the remote control and/or interaction-with electronic-devices such as computers; home-entertainment-systems; media-centers; televisions; DVD-players; VCR-players; music systems; appliances; security systems; toys/games; and/or displays are described in U.S. Pat. No. 8,508,472 to Wieder entitled: “Wearable remote control with a single control button”, which is incorporated in its entirety for all purposes as if fully set forth herein. A user may orient a pointer (e.g., laser pointer) to place a pointer-spot on/near object(s) on an active-display(s); and/or a fixed-display(s); and/or on real-world object(s) within a display region or pointer-spot detection-region. Detectors, imager(s) and/or camera(s) may be connected/attached to the display region and/or a structure that is connected/attached to display region. When the user initiates a “select”, the detectors/cameras may detect the location of the pointer-spot within the display region. Corresponding to the user's selection(s); control action(s) may be performed on the device(s) being controlled/interacted-with and additional selection-menus may be optionally presented on an active-display.

A hand-worn controller consisting of a housing having a central opening sized to permit the controller to be worn as ring on the index finger of a human hand is described in U.S. Patent Application Publication No. 2006/0164383 to Machin et al. entitled: “Remote controller ring for user interaction”, which is incorporated in its entirety for all purposes as if fully set forth herein. A joystick lever projects outwardly from said housing and is positioned to be manipulated by the user's thumb. The joystick operates on or more control devices, such as switches or potentiometers, that produce control signals. A wireless communications device, such as a Bluetooth module, mounted in said housing transmits command signals to a remote utilization device, which are indicative of the motion or position of said joystick lever.

A wearable augmented reality computing apparatus with a display screen, a reflective device, a computing device and a head mounted harness to contain these components is described in U.S. Patent Application Publication No. 2012/0050144 to Morlock entitled: “Wearable augmented reality computing apparatus”, which is incorporated in its entirety for all purposes as if fully set forth herein. The display device and reflective device are configured such that a user can see the reflection from the display device superimposed on the view of reality. An embodiment uses a switchable mirror as the reflective device. One usage of the apparatus is for vehicle or pedestrian navigation. The portable display and general purpose computing device can be combined in a device such as a smartphone. Additional components consist of orientation sensors and non-handheld input devices.

In one example, a wearable device may use, or may be based on, a processor or a microcontroller that is designed for wearable applications, such as the CC2650 SimpleLink™ Multistandard Wireless MCU available from Texas Instruments Incorporated (headquartered in Dallas, Tex., U.S.A.) and described in a Texas Instrument 2015 publication #SWRT022 entitled: “ SimpleLink™ Ultra - Low Power—Wireless Microcontroller Platform ”, and in a Texas Instrument 2015 datasheet #SWRS158A (published February 2015, Revised October 2015) entitled: “CC2650 SimpleLink™ Multistandard Wireless MCU ”, which are both incorporated in their entirety for all purposes as if fully set forth herein.

An example of a personal multimedia electronic device, and more particularly to a head-worn device such as an eyeglass frame, is described in U.S. Patent Application No. 2010/0110368 to Chaum entitled: “System and apparatus for eyeglass appliance platform”, which is incorporated in its entirety for all purposes as if fully set forth herein. The device is having a plurality of interactive electrical/optical components. In one embodiment, a personal multimedia electronic device includes an eyeglass frame having a side arm and an optic frame; an output device for delivering an output to the wearer; an input device for obtaining an input; and a processor comprising a set of programming instructions for controlling the input device and the output device. The output device is supported by the eyeglass frame and is selected from the group consisting of a speaker, a bone conduction transmitter, an image projector, and a tactile actuator. The input device is supported by the eyeglass frame and is selected from the group consisting of an audio sensor, a tactile sensor, a bone conduction sensor, an image sensor, a body sensor, an environmental sensor, a global positioning system receiver, and an eye tracker. In one embodiment, the processor applies a user interface logic that determines a state of the eyeglass device and determines the output in response to the input and the state.

An example of an eyewear for a user is described in U.S. Patent Application No. 2012/0050668 Howell et al. entitled: “Eyewear with touch-sensitive input surface”, which is incorporated in its entirety for all purposes as if fully set forth herein. The eyewear includes an eyewear frame, electrical circuitry at least partially in the eyewear frame, and a touch sensitive input surface on the eyewear frame configured to provide an input to the electrical circuitry to perform a function via touching the touch sensitive input surface. In another embodiment, the eyewear includes a switch with at least two operational states. The operational states of the switch can be configured to be changed by sliding a finger across the touch sensitive input surface of the frame.

An example of a wearable computing device is described in U.S. Patent Application No. 2013/0169513 to Heinrich et al. entitled: “Wearable computing device”, which is incorporated in its entirety for all purposes as if fully set forth herein. The device includes a bone conduction transducer, an extension arm, a light pass hole, and a flexible touch pad input circuit. When a user wears the device, the transducer contacts the user's head. A display is attached to a free end of an extension arm. The extension arm is pivotable such that a distance between the display and the user's eye is adjustable to provide the display at an optimum position. The light pass hole may include a light emitting diode and a flash. The touch pad input circuit may be adhered to at least one side arm such that parting lines are not provided between edges of the circuit and the side arm.

Speech synthesis. A speech synthesizer is used to produce natural and intelligible artificial human speech may be implemented in hardware, in software, or combination thereof. A speech synthesizer may be Text-To-Speech (TTS) based, that converts normal language text to speech, or alternatively (or in addition) may be based on rendering symbolic linguistic representation like phonetic transcription. A TTS typically involves two steps, the front-end where the raw input text is pre-processed to fully write-out words replacing numbers and abbreviations, followed by assigning phonetic transcriptions to each word (text-to-phoneme), and the back-end (or synthesizer) where the symbolic linguistic representation is converted to output sound.

The generating of synthetic speech waveform typically uses a concatenative or formant synthesis. The concatenative synthesis commonly produces the most natural-sounding synthesized speech, and is based on the concatenation (or stringing together) of segments of recorded speech. There are three main types of concatenative synthesis: Unit selection, diphone synthesis, and domain-specific synthesis. Unit selection synthesis is based on large databases of recorded speech including individual phones, diphones, half-phones, syllables, morphemes, words, phrases, and sentences, indexed based on the segmentation and acoustic parameters like the fundamental frequency (pitch), duration, position in the syllable, and neighboring phones. At run time, the desired target utterance is created by determining (typically using a specially weighted decision tree) the best chain of candidate units from the database (unit selection). Diphone synthesis uses a minimal speech database containing all the diphones (sound-to-sound transitions) occurring in a language, and at runtime, the target prosody of a sentence is superimposed on these minimal units by means of digital signal processing techniques such as linear predictive coding. Domain-specific synthesis is used where the output is limited to a particular domain, using concatenated prerecorded words and phrases to create complete utterances. In formant synthesis the synthesized speech output is created using additive synthesis and an acoustic model (physical modeling synthesis), rather than on using human speech samples. Parameters such as fundamental frequency, voicing, and noise levels are varied over time to create a waveform of artificial speech. The synthesis may further be based on articulatory synthesis where computational techniques for synthesizing speech are based on models of the human vocal tract and the articulation processes occurring there, or may be HMM-based synthesis which is based on hidden Markov models, where the frequency spectrum (vocal tract), fundamental frequency (vocal source), and duration (prosody) of speech are modeled simultaneously by HMMs and generated based on the maximum likelihood criterion. The speech synthesizer may further be based on the book entitled: “ Development in Speech Synthesis ”, by Mark Tatham and Katherine Morton, published 2005 by John Wiley & Sons Ltd., ISBN: 0-470-85538-X, on the book entitled: “ Speech Synthesis and Recognition ” by John Holmes and Wendy Holmes, 2 nd Edition, published 2001 ISBN: 0-7484-0856-8, on the book entitled: “ Techniques and Challenges in Speech Synthesis—Final Report ” by David Ferris [ELEC4840B] published Apr. 11, 2016, and on the book entitled: “Text-to-Speech Synthesis” by Paul Taylor [ISBN 978-0-521-89927-7] published 2009 by Cambridge University Press, which are all incorporated in their entirety for all purposes as if fully set forth herein.

A speech synthesizer may be software-based such as Apple VoiceOver utility which uses speech synthesis for accessibility, and is part of the Apple iOS operating system used on the iPhone, iPad and iPod Touch. Similarly, Microsoft uses SAPI 4.0 and SAPI 5.0 as part of Windows operating system. A speech synthesizer may be hardware based, such as based on Sensory Inc. NLP-5x described in the Data sheet “ Natural Language Processor with Motor, Sensor and Display Control ”, P/N 80-0317-K, published 2010 by Sensory, Inc. of Santa-Clara, Calif., U.S.A., which is incorporated herein in its entirety for all purposes as if fully set forth herein.

In one example, the sounder may be used to play a stored digital audio. The digital audio content can be stored in the sounder. Further, few files may be stored (e.g., representing different announcements or songs), selected by the control logic. Alternatively or in addition, the digital audio data may be received by the sounder from external sources via any of the above networks. Furthermore, the source of the digital audio may be a microphone serving as a sensor, either after processing, storing, delaying, or any other manipulation, or as originally received resulting ‘doorphone’ or ‘intercom’ functionality between a microphone and a sounder in the building.

In another example, the sounder simulates the voice of a human being or generates music, typically by using an electronic circuit having a memory for storing the sounds (e.g., music, song, voice message, etc.), a digital to analog converter 62 to reconstruct the electrical representation of the sound, and a driver for driving a loudspeaker, which is an electro-acoustic transducer that converts an electrical signal to sound. An example of a greeting card providing music and mechanical movement is disclosed in U.S. Patent Application No. 2007/0256337 to Segan entitled: “User Interactive Greeting Card”, which is incorporated in its entirety for all purposes as if fully set forth herein.

In one example, the system is used for sound or music generation. For example, the sound produced can emulate the sounds of a conventional acoustical music instrument, such as a piano, tuba, harp, violin, flute, guitar and so forth. In one example, the sounder is an audible signaling device, emitting audible sounds that can be heard (having frequency components in the 20-20,000 Hz band). In one example the sound generated is music or song. The elements of the music such as pitch (which governs melody and harmony), rhythm (and its associated concepts tempo, meter, and articulation), dynamics, and the sonic qualities of timbre and texture, may be associated with the shape theme. For example, if a musical instrument shown in the picture, the music generated by that instrument will be played, e.g., drumming sound of drums and playing of a flute or guitar. In one example, a talking human voice is played by the sounder. The sound may be a syllable, a word, a phrase, a sentence, a short story or a long story, and can be based on speech synthesis or prerecorded. Male or female voice can be used, further being young or old.

Some examples of toys that include generation of an audio signal such as music are disclosed in U.S. Pat. No. 4,496,149 to Schwartzberg entitled: “Game Apparatus Utilizing Controllable Audio Signals”, in U.S. Pat. No. 4,516,260 to Breedlove et al. entitled: “Electronic Learning Aid or Game having Synthesized Speech”, in U.S. Pat. No. 7,414,186 to Scarpa et al. entitled: “System and Method for Teaching Musical Notes”, in U.S. Pat. No. 4,968,255 to Lee et al., entitled: “Electronic Instructional Apparatus”, in U.S. Pat. No. 4,248,123 to Bunger et al., entitled: “Electronic Piano” and in U.S. Pat. No. 4,796,891 to Milner entitled: “Musical Puzzle Using Sliding Tiles”, and toys with means for synthesizing human voice are disclosed in U.S. Pat. No. 6,527,611 to Cummings entitled: “Place and Find Toy”, and in U.S. Pat. No. 4,840,602 to Rose entitled: “Talking Doll Responsive to External Signal”, which are all incorporated in their entirety for all purposes as if fully set forth herein. A music toy kit combining music toy instrument with a set of construction toy blocks is disclosed in U.S. Pat. No. 6,132,281 to Klitsner et al. entitled: “Music Toy Kit” and in U.S. Pat. No. 5,349,129 to Wisniewski et al. entitled: “Electronic Sound Generating Toy”, which are incorporated in their entirety for all purposes as if fully set forth herein.

Database. A database is an organized collection of data, typically managed by a DataBase Management System (DBMS) that organizes the storage of data and performs other functions such as the creation, maintenance, and usage of the database storage structures. The data is typically organized to model aspects of reality in a way that supports processes requiring information. Databases commonly also provide users with a user interface and front-end that enables the users to query the database, often in complex manners that require processing and organization of the data. The term “database” is used herein to refer to a database, or to both a database and the DBMS used to manipulate it. Database Management Systems (DBMS) are typically computer software applications that interact with the user, other applications, and the database itself to capture and analyze data, typically providing various functions that allow entry, storage and retrieval of large quantities of information, as well as providing ways to manage how that information is organized. A general-purpose DBMS is designed to allow the definition, creation, querying, update, and administration of databases. Examples of DBMSs include MySQL, PostgreSQL, Microsoft SQL Server, Oracle, Sybase and IBM DB2. Database technology and application is described in a document published by Telemark University College entitled “Introduction to Database Systems”, authored by Hans-Petter Halvorsen (dated 2014 Mar. 3), which is incorporated in its entirety for all purposes as if fully set forth herein.

SQL. Structured Query Language (SQL) is a widely-used programming language for working with relational databases, designed for managing data held in a relational database management system (RDBMS), or for stream processing in a relational data stream management system (RDSMS). SQL consists of a data definition language and a data manipulation language. The scope of SQL includes data insert, query, update and delete, schema creation and modification, and data access control. Although SQL is often described as, and largely is, a declarative language ( 4 GL), it also includes procedural elements. SQL is designed for querying data contained in a relational database, and is a set-based, declarative query language. The SQL is standardized as ISO/IEC 9075:2011 standard: “ Information technology—Database languages—SQL ”. The ISO/IEC 9075 standard is complemented by ISO/IEC 13249 standard: “SQL Multimedia and Application Packages” that defines interfaces and packages based on SQL. The aim is a unified access to typical database applications like text, pictures, data mining or spatial data. SQL is described in the tutorial entitled: “ Oracle/SQL Tutorial ” by Michael Gertz of University of California, which is incorporated in its entirety for all purposes as if fully set forth herein.

DSP. A Digital Signal Processor (DSP) is a specialized microprocessor (or a SIP block), with its architecture optimized for the operational needs of digital signal processing, serving the goal of DSPs is usually to measure, filter and/or compress continuous real-world analog signals. Most general-purpose microprocessors can also execute digital signal processing algorithms successfully, but dedicated DSPs usually have better power efficiency thus they are more suitable in portable devices such as mobile phones because of power consumption constraints. DSPs often use special memory architectures that are able to fetch multiple data and/or instructions at the same time. Digital signal processing algorithms typically require a large number of mathematical operations to be performed quickly and repeatedly on a series of data samples. Signals (perhaps from audio or video sensors) are constantly converted from analog to digital, manipulated digitally, and then converted back to analog form. Many DSP applications have constraints on latency; that is, for the system to work, the DSP operation must be completed within some fixed time, and deferred (or batch) processing is not viable. A specialized digital signal processor, however, will tend to provide a lower-cost solution, with better performance, lower latency, and no requirements for specialized cooling or large batteries. The architecture of a digital signal processor is optimized specifically for digital signal processing. Most also support some of the features as an applications processor or microcontroller, since signal processing is rarely the only task of a system. Some useful features for optimizing DSP algorithms are outlined below.

Hardware features visible through DSP instruction sets commonly include hardware modulo addressing, allowing circular buffers to be implemented without having to constantly test for wrapping; a memory architecture designed for streaming data, using DMA extensively and expecting code to be written to know about cache hierarchies and the associated delays; driving multiple arithmetic units may require memory architectures to support several accesses per instruction cycle; separate program and data memories (Harvard architecture), and sometimes concurrent access on multiple data buses; and specianalyzeal SIMD (single instruction, multiple data) operations. Digital signal processing is further described in a book by John G. Proakis and Dimitris G. Manolakis, published 1996 by Prentice-Hall Inc. [ISBN 0-13-394338-9] entitled: “ Third Edition—DIGITAL SIGNAL PROCESSING—Principles, Algorithms, and Application ”, and in a book by Steven W. Smith entitled: “ The Scientist and Engineer's Guide to—Digital Signal Processing—Second Edition ”, published by California Technical Publishing [ISBN 0-9960176-7-6], which are both incorporated in their entirety for all purposes as if fully set forth herein.

ANN. Neural networks (or Artificial Neural Networks (ANNs)) are a family of statistical learning models inspired by biological neural networks (the central nervous systems of animals, in particular the brain) and are used to estimate or approximate functions that may depend on a large number of inputs and are generally unknown. Artificial neural networks are generally presented as systems of interconnected “neurons” which send messages to each other. The connections have numeric weights that can be tuned based on experience, making neural nets adaptive to inputs and capable of learning. For example, a neural network for handwriting recognition is defined by a set of input neurons that may be activated by the pixels of an input image. After being weighted and transformed by a function (determined by the network designer), the activations of these neurons are then passed on to other neurons, and this process is repeated until finally, an output neuron is activated, and determines which character was read. Like other machine learning methods—systems that learn from data—neural networks have been used to solve a wide variety of tasks that are hard to solve using ordinary rule-based programming, including computer vision and speech recognition. A class of statistical models is typically referred to as “Neural” if it contains sets of adaptive weights, i.e. numerical parameters that are tuned by a learning algorithm, and capability of approximating non-linear functions from their inputs. The adaptive weights can be thought of as connection strengths between neurons, which are activated during training and prediction. Neural Networks are described in a book by David Kriesel entitled: “A Brief Introduction to Neural Networks” (ZETA2-EN) [downloaded May 2015 from www.dkriesel.com], which is incorporated in its entirety for all purposes as if fully set forth herein. Neural Networks are further described in a book by Simon Haykin published 2009 by Pearson Education, Inc. [ISBN-978-0-13-147139-9] entitled: “ Neural Networks and Learning Machines—Third Edition ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Neural networks based techniques may be used for image processing, as described in an article in Engineering Letters, 20:1, EL_20_1_09 (Advance online publication: 27 Feb. 2012) by Juan A. Ramirez-Quintana, Mario I. Cacon-Murguia, and F. Chacon-Hinojos entitled: “ Artificial Neural Image Processing Applications: A Survey ”, in an article published 2002 by Pattern Recognition Society in Pattern Recognition 35 (2002) 2279-2301 [PII: S0031-3203(01)00178-9] authored by M. Egmont-Petersen, D. de Ridder, and H. Handels entitled: “ Image processing with neural networks—a review ”, and in an article by Dick de Ridder et al. (of the Utrecht University, Utrecht, The Netherlands) entitled: “ Nonlinear image processing using artificial neural networks ”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

Neural networks may be used for object detection as described in an article by Christian Szegedy, Alexander Toshev, and Dumitru Erhan (of Google, Inc.) (downloaded July 2015) entitled: “ Deep Neural Networks for Object Detection ”, in a CVPR2014 paper provided by the Computer Vision Foundation by Dumitru Erhan, Christian Szegedy, Alexander Toshev, and Dragomir Anguelov (of Google, Inc., Mountain-View, Calif., U.S.A.) (downloaded July 2015) entitled: “ Scalable Object Detection using Deep Neural Networks ”, and in an article by Shawn McCann and Jim Reesman (both of Stanford University) (downloaded July 2015) entitled: “Object Detection using Convolutional Neural Networks”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

Using neural networks for object recognition or classification is described in an article (downloaded July 2015) by Mehdi Ebady Manaa, Nawfal Turki Obies, and Dr. Tawfiq A. Al-Assadi (of Department of Computer Science, Babylon University), entitled: “ Object Classification using neural networks with Gray - level Co - occurrence Matrices ( GLCM )”, in a technical report No. IDSIA-01-11 January 2001 published by IDSIA/USI-SUPSI and authored by Dan C. Ciresan et al. entitled: “ High - Performance Neural Networks for Visual Object Classification ”, in an article by Yuhua Zheng et al. (downloaded July 2015) entitled: “ Object Recognition using Neural Networks with Bottom - Up and top - Down Pathways ”, and in an article (downloaded July 2015) by Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman (all of Visual Geometry Group, University of Oxford), entitled: “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

Using neural networks for object recognition or classification is further described in U.S. Pat. No. 6,018,728 to Spence et al. entitled: “Method and Apparatus for Training a Neural Network to Learn Hierarchical Representations of Objects and to Detect and Classify Objects with Uncertain Training Data”, in U.S. Pat. No. 6,038,337 to Lawrence et al. entitled: “Method and Apparatus for Object Recognition”, in U.S. Pat. No. 8,345,984 to Ji et al. entitled: “3D Convolutional Neural Networks for Automatic Human Action Recognition”, and in U.S. Pat. No. 8,705,849 to Prokhorov entitled: “Method and System for Object Recognition Based on a Trainable Dynamic System”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

Actual ANN implementation may be based on, or may use, the MATLB® ANN described in the User's Guide Version 4 published July 2002 by The MathWorks, Inc. (Headquartered in Natick, Mass., U.S.A.) entitled: “ Neural Network ToolBox—For Use with MATLAB ®” by Howard Demuth and Mark Beale, which is incorporated in its entirety for all purposes as if fully set forth herein. A VHDL IP core that is a configurable feedforward Artificial Neural Network (ANN) for implementation in FPGAs is available (under the Name: artificial_neural_network, created Jun. 2, 2016 and updated Oct. 11, 2016) from OpenCores organization, downloadable from http://opencores.org/. This IP performs full feedforward connections between consecutive layers. All neurons' outputs of a layer become the inputs for the next layer. This ANN architecture is also known as Multi-Layer Perceptron (MLP) when is trained with a supervised learning algorithm. Different kinds of activation functions can be added easily coding them in the provided VHDL template. This IP core is provided in two parts: kernel plus wrapper. The kernel is the optimized ANN with basic logic interfaces. The kernel should be instantiated inside a wrapper to connect it with the user's system buses. Currently, an example wrapper is provided for instantiate it on Xilinx Vivado, which uses AXI4 interfaces for AMBA buses.

Dynamic neural networks are the most advanced in that they dynamically can, based on rules, form new connections and even new neural units while disabling others. In a feedforward neural network (FNN), the information moves in only one direction—forward: From the input nodes data goes through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network. Feedforward networks can be constructed from different types of units, e.g. binary McCulloch-Pitts neurons, the simplest example being the perceptron. Contrary to feedforward networks, Recurrent Neural Networks (RNNs) are models with bi-directional data flow. While a feedforward network propagates data linearly from input to output, RNNs also propagate data from later processing stages to earlier stages. RNNs can be used as general sequence processors.

Any ANN herein may be based on, may use, or may be trained or used, using the schemes, arrangements, or techniques described in the book by David Kriesel entitled: “A Brief Introduction to Neural Networks” (ZETA2-EN) [downloaded May 2015 from www.dkriesel.com], in the book by Simon Haykin published 2009 by Pearson Education, Inc. [ISBN-978-0-13-147139-9] entitled: “ Neural Networks and Learning Machines—Third Edition ”, in the article in Engineering Letters, 20:1, EL_20_1_09 (Advance online publication: 27 Feb. 2012) by Juan A. Ramirez-Quintana, Mario I. Cacon-Murguia, and F. Chacon-Hinojos entitled: “ Artificial Neural Image Processing Applications: A Survey ”, or in the article entitled: “ Image processing with neural networks—a review ”, and in the article by Dick de Ridder et al. (of the Utrecht University, Utrecht, The Netherlands) entitled: “Nonlinear image processing using artificial neural networks”.

Any object detection herein using ANN may be based on, may use, or may be trained or used, using the schemes, arrangements, or techniques described in the article by Christian Szegedy, Alexander Toshev, and Dumitru Erhan (of Google, Inc.) entitled: “ Deep Neural Networks for Object Detection ”, in the CVPR2014 paper provided by the Computer Vision Foundation entitled: “Scalable Object Detection using Deep Neural Networks”, in the article by Shawn McCann and Jim Reesman entitled: “ Object Detection using Convolutional Neural Networks ”, or in any other document mentioned herein.

Any object recognition or classification herein using ANN may be based on, may use, or may be trained or used, using the schemes, arrangements, or techniques described in the article by Mehdi Ebady Manaa, Nawfal Turki Obies, and Dr. Tawfiq A. Al-Assadi entitled: “ Object Classification using neural networks with Gray - level Co - occurrence Matrices ( GLCM )”, in the technical report No. IDSIA-01-11 entitled: “ High - Performance Neural Networks for Visual Object Classification ”, in the article by Yuhua Zheng et al. entitled: “ Object Recognition using Neural Networks with Bottom - Up and top - Down Pathways ”, in the article by Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, entitled: “ Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps ”, or in any other document mentioned herein.

A logical representation example of a simple feed-forward Artificial Neural Network (ANN) 40 is shown in FIG. 4 . The ANN 40 provides three inputs designated as IN #1 42 a , IN #2 42 b , and IN #3 42 c , which connects to three respective neuron units forming an input layer 41 a . Each neural unit is linked some of, or to all of, a next layer 41 b , with links that may be enforced or inhibit by associating weights as part of the training process. An output layer 41 d consists of two neuron units that feeds two outputs OUT #1 43 a and OUT #2 43 b . Another layer 41 c is coupled between the layer 41 b and the output layer 41 d . The intervening layers 41 b and 41 c are referred to as hidden layers. While three inputs are exampled in the ANN 40 , any number of inputs may be equally used, and while two output are exampled in the ANN 40 , any number of outputs may equally be used. Further, the ANN 40 uses four layers, consisting of an input layer, an output layer, and two hidden layers. However, any number of layers may be used. For example, the number of layers may be equal to, or above than, 3, 4, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers. Similarly, an ANN may have any number below 4, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers.

DNN. A Deep Neural Network (DNN) is an artificial neural network (ANN) with multiple layers between the input and output layers. For example, a DNN that is trained to recognize dog breeds will go over the given image and calculate the probability that the dog in the image is a certain breed. The user can review the results and select which probabilities the network should display (above a certain threshold, etc.) and return the proposed label. Each mathematical manipulation as such is considered a layer, and complex DNN have many layers, hence the name “deep” networks. DNNs can model complex non-linear relationships. DNN architectures generate compositional models where the object is expressed as a layered composition of primitives. The extra layers enable composition of features from lower layers, potentially modeling complex data with fewer units than a similarly performing shallow network. Deep architectures include many variants of a few basic approaches. Each architecture has found success in specific domains. It is not always possible to compare the performance of multiple architectures, unless they have been evaluated on the same data sets. DNN is described in a book entitled: “ Introduction to Deep Learning From Logical Calculus to Artificial Intelligence ” by Sandro Skansi [ISSN 1863-7310 ISSN 2197-1781, ISBN 978-3-319-73003-5], published 2018 by Springer International Publishing AG, which is incorporated in its entirety for all purposes as if fully set forth herein.

Deep Neural Networks (DNNs), which employ deep architectures can represent functions with higher complexity if the numbers of layers and units in a single layer are increased. Given enough labeled training datasets and suitable models, deep learning approaches can help humans establish mapping functions for operation convenience. In this paper, four main deep architectures are recalled and other methods (e.g. sparse coding) are also briefly discussed. Additionally, some recent advances in the field of deep learning are described. The purpose of this article is to provide a timely review and introduction on the deep learning technologies and their applications. It is aimed to provide the readers with a background on different deep learning architectures and also the latest development as well as achievements in this area. The rest of the paper is organized as follows. In Sections II-V, four main deep learning architectures, which are Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), AutoEncoder (AE), and Convolutional Neural Networks (CNNs), are reviewed, respectively. Comparisons are made among these deep architectures and recent developments on these algorithms are discussed. A schematic diagram 40 a of an RBM, a schematic diagram 40 b of a DBN, and a schematic structure 40 c of a CNN are shown in FIG. 4 a.

DNNs are typically feedforward networks in which data flows from the input layer to the output layer without looping back. At first, the DNN creates a map of virtual neurons and assigns random numerical values, or “weights”, to connections between them. The weights and inputs are multiplied and return an output between 0 and 1. If the network did not accurately recognize a particular pattern, an algorithm would adjust the weights. That way the algorithm can make certain parameters more influential, until it determines the correct mathematical manipulation to fully process the data. Recurrent neural networks (RNNs), in which data can flow in any direction, are used for applications such as language modeling. Long short-term memory is particularly effective for this use. Convolutional deep neural networks (CNNs) are used in computer vision. CN Ns also have been applied to acoustic modeling for Automatic Speech Recognition (ASR).

Since the proposal of a fast learning algorithm for deep belief networks in 2006, the deep learning techniques have drawn ever-increasing research interests because of their inherent capability of overcoming the drawback of traditional algorithms dependent on hand-designed features. Deep learning approaches have also been found to be suitable for big data analysis with successful applications to computer vision, pattern recognition, speech recognition, natural language processing, and recommendation systems.

Widely-used deep learning architectures and their practical applications are discussed in a paper entitled: “ A Survey of Deep Neural Network Architectures and Their Applications ” by Weibo Liva, Zidong Wanga, Xiaohui Liva, Nianyin Zengb, Yurong Liuc, and Fuad E. Alsaadid, published December 2016 [DOI: 10.1016/j.neucom.2016.12.038] in Neurocomputing 234, which is incorporated in its entirety for all purposes as if fully set forth herein. An up-to-date overview is provided on four deep learning architectures, namely, autoencoder, convolutional neural network, deep belief network, and restricted Boltzmann machine. Different types of deep neural networks are surveyed and recent progresses are summarized. Applications of deep learning techniques on some selected areas (speech recognition, pattern recognition and computer vision) are highlighted. A list of future research topics are finally given with clear justifications.

RBM. Restricted Boltzmann machine (RBM) is a generative stochastic artificial neural network that can learn a probability distribution over its set of inputs. As their name implies, RBMs are a variant of Boltzmann machines, with the restriction that their neurons must form a bipartite graph: a pair of nodes from each of the two groups of units (commonly referred to as the “visible” and “hidden” units respectively) may have a symmetric connection between them; and there are no connections between nodes within a group. By contrast, “unrestricted” Boltzmann machines may have connections between hidden units. This restriction allows for more efficient training algorithms than are available for the general class of Boltzmann machines, in particular the gradient-based contrastive divergence algorithm. Restricted Boltzmann machines can also be used in deep learning networks. In particular, deep belief networks can be formed by “stacking” RBMs and optionally fine-tuning the resulting deep network with gradient descent and backpropagation

DBN. A Deep Belief Network (DBN) is a generative graphical model, or alternatively a class of deep neural network, composed of multiple layers of latent variables (“hidden units”), with connections between the layers but not between units within each layer. When trained on a set of examples without supervision, a DBN can learn to probabilistically reconstruct its inputs. The layers then act as feature detectors. After this learning step, a DBN can be further trained with supervision to perform classification. DBNs can be viewed as a composition of simple, unsupervised networks such as restricted Boltzmann machines (RBMs) or autoencoders, where each sub-network's hidden layer serves as the visible layer for the next. An RBM is an undirected, generative energy-based model with a “visible” input layer and a hidden layer and connections between but not within layers. This composition leads to a fast, layer-by-layer unsupervised training procedure, where contrastive divergence is applied to each sub-network in turn, starting from the “lowest” pair of layers (the lowest visible layer is a training set).

Dynamic neural networks are the most advanced in that they dynamically can, based on rules, form new connections and even new neural units while disabling others. In a Feedforward Neural Network (FNN), the information moves in only one direction—forward: From the input nodes data goes through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network. Feedforward networks can be constructed from different types of units, e.g. binary McCulloch-Pitts neurons, the simplest example being the perceptron. Contrary to feedforward networks, Recurrent Neural Networks (RNNs) are models with bi-directional data flow. While a feedforward network propagates data linearly from input to output, RNNs also propagate data from later processing stages to earlier stages. RNNs can be used as general sequence processors.

A waveform analysis assembly (10) that includes a sensor (12) for detecting physiological electrical and mechanical signals produced by the body is disclosed in U.S. Pat. No. 5,092,343 to Spitzer et al. entitled: “Waveform analysis apparatus and method using neural network techniques”, which is incorporated in its entirety for all purposes as if fully set forth herein. An extraction neural network (22, 22′) will learn a repetitive waveform of the electrical signal, store the waveform in memory (18), extract the waveform from the electrical signal, store the location times of occurrences of the waveform, and subtract the waveform from the electrical signal. Each significantly different waveform in the electrical signal is learned and extracted. A single or multilayer layer neural network (22, 22′) accomplishes the learning and extraction with either multiple passes over the electrical signal or accomplishes the learning and extraction of all waveforms in a single pass over the electrical signal. A reducer (20) receives the stored waveforms and times and reduces them into features characterizing the waveforms. A classifier neural network (36) analyzes the features by classifying them through non-linear mapping techniques within the network representing diseased states and produces results of diseased states based on learned features of the normal and patient groups.

A real-time waveform analysis system that utilizes neural networks to perform various stages of the analysis is disclosed in U.S. Pat. No. 5,751,911 to Goldman entitled: “Real-time waveform analysis using artificial neural networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. The signal containing the waveform is first stored in a buffer and the buffer contents transmitted to a first and second neural network, which have been previously trained to recognize the start point and the end point of the waveform respectively. A third neural network receives the signal occurring between the start and end points and classifies that waveform as comprising either an incomplete waveform, a normal waveform or one of a variety of predetermined characteristic classifications. Ambiguities in the output of the third neural network are arbitrated by a fourth neural network, which may be given additional information, which serves to resolve these ambiguities. In accordance with the preferred embodiment, the present invention is applied to a system analyzing respiratory waveforms of a patient undergoing anesthesia and the classifications of the waveform correspond to normal or various categories of abnormal features functioning in the respiratory signal. The system performs the analysis rapidly enough to be used in real-time systems and can be operated with relatively low-cost hardware and with minimal software development required.

A method for analyzing data is disclosed in U.S. Pat. No. 8,898,093 to Helmsen entitled: “Systems and methods for analyzing data using deep belief networks (DBN) and identifying a pattern in a graph”, which is incorporated in its entirety for all purposes as if fully set forth herein. The method includes generating, using a processing device, a graph from raw data, the graph including a plurality of nodes and edges, deriving, using the processing device, at least one label for each node using a deep belief network, and identifying, using the processing device, a predetermined pattern in the graph based at least in part on the labeled nodes.

Signal Analysis. Digital Signal Processing (DSP) is the use of digital processing, such as by computers, to perform a wide variety of signal processing operations. The signals processed in this manner are a sequence of numbers that represent samples of a continuous variable in a domain such as time, space, or frequency. Digital signal processing and analog signal processing are subfields of signal processing. DSP applications include audio and speech signal processing, sonar, radar and other sensor array processing, spectral estimation, statistical signal processing, digital image processing, signal processing for telecommunications, control of systems, biomedical engineering, seismic data processing, among others. Digital signal processing can involve linear or nonlinear operations. Nonlinear signal processing is closely related to nonlinear system identification and can be implemented in the time, frequency, and spatio-temporal domains.

In DSP, digital signals are analyzed in one of the following domains: time domain (one-dimensional signals), spatial domain (multidimensional signals), frequency domain, and wavelet domains. The domain in which to process a signal is determined by making an informed assumption (or by trying different possibilities) as to which domain best represents the essential characteristics of the signal. A sequence of samples from a measuring device produces a temporal or spatial domain representation, whereas a discrete Fourier transform produces the frequency domain information, that is, the frequency spectrum. Signal analysis is further described in Agilent Technologies Application Note 243 published 2000 [5952-8898E) entitled: “ The Fundamentals of Signal Processing ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Time domain is the analysis of mathematical functions, physical signals or time series of economic or environmental data, with respect to time. In the time domain, the signal or function's value is known for all real numbers, for the case of continuous time, or at various separate instants in the case of discrete time. An oscilloscope is a tool commonly used to visualize real-world signals in the time domain. A time-domain graph shows how a signal changes with time, whereas a frequency-domain graph shows how much of the signal lies within each given frequency band over a range of frequencies.

In frequency domain analysis, also known as spectrum- or spectral analysis, Signals are converted from time or space domain to the frequency domain usually through the Fourier transform. The Fourier transform converts the signal information to a magnitude and phase component of each frequency. Often the Fourier transform is converted to the power spectrum, which is the magnitude of each frequency component squared. The most common purpose for analysis of signals in the frequency domain is analysis of signal properties. The engineer can study the spectrum to determine which frequencies are present in the input signal and which are missing. There are some commonly used frequency domain transformations. For example, the cepstrum converts a signal to the frequency domain through Fourier transform, takes the logarithm, then applies another Fourier transform. This emphasizes the harmonic structure of the original spectrum. Fourier Transform is described in Lectures Notes entitled: “ EE 261 —The Fourier Transform and its Applications ” by Prof. Brad Osgood of the Electrical Engineering Department, Stanford University, downloaded from the Internet on November 2016, which is incorporated in its entirety for all purposes as if fully set forth herein.

A spectrum analyzer measures the magnitude of an input signal versus frequency within the full frequency range of the instrument. The primary use is to measure the power of the spectrum of known and unknown signals. The input signal that a spectrum analyzer measures is electrical; however, spectral compositions of other signals, such as acoustic pressure waves and optical light waves, can be considered through the use of an appropriate transducer. By analyzing the spectra of electrical signals, dominant frequency, power, distortion, harmonics, bandwidth, and other spectral components of a signal can be observed that are not easily detectable in time domain waveforms. These parameters are useful in the characterization of electronic devices, such as wireless transmitters.

Spectrum analyzer types are distinguished by the methods used to obtain the spectrum of a signal. There are swept-tuned and Fast Fourier Transform (FFT) based spectrum analyzers. A swept-tuned analyzer uses a superheterodyne receiver to down-convert a portion of the input signal spectrum to the center frequency of a narrow band-pass filter, whose instantaneous output power is recorded or displayed as a function of time. By sweeping the receiver's center-frequency (using a voltage-controlled oscillator) through a range of frequencies, the output is also a function of frequency. While the sweep centers on any particular frequency, it may be missing short-duration events at other frequencies. An FFT analyzer computes a time-sequence of periodograms. FFT refers to a particular mathematical algorithm used in the process. This is commonly used in conjunction with a receiver and analog-to-digital converter. As above, the receiver reduces the center-frequency of a portion of the input signal spectrum, but the portion is not swept. The purpose of the receiver is to reduce the sampling rate that is contended by the analyzer. With a sufficiently low sample-rate, FFT analyzers can process all the samples (100% duty-cycle), and are therefore able to avoid missing short-duration events. Spectrum analyzer basics are described in Agilent Technologies Application Note 150 published Feb. 25, 2014 [5952-0292] entitled: “ Spectrum Analysis Basics ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Audio signal processing. Audio signal processing, sometimes referred to as audio processing, is the intentional alteration of auditory signals, or sound, often through an audio effect or effects unit. As audio signals may be electronically represented in either digital or analog format, signal processing may occur in either domain. Analog processors operate directly on the electrical signal, while digital processors operate mathematically on the digital representation of that signal. Audio signal processing is described in a book published 2003 by David Rocchesso (Universita di Verona) entitled: “ Introduction to Sound Processing ” [ISBN 88-901126-1-1], and in a book by Udo Zolzer of the Technical University of Hamburg-Harburg, Germany published 1995 by John Wiley & Sons, Ltd. [ISBN 0-47197226-6] entitled: “ Digital audio Signal Processing ”, which are both incorporated in their entirety for all purposes as if fully set forth herein. An example of a digital audio processor is IC model TDA7590 available from STMicroelectronics NV, described in a data sheet Rev. 3 entitled: “ TDA 7590 —Digital signal processing IC for speech and audio applications ” published 2013 by STMicroelectronics, which is incorporated in its entirety for all purposes as if fully set forth herein. Another example of DSP IC is Model No. TMS320C6678 available from Texas Instruments Incorporated, headquartered in Dallas, Tex., U.S.A., described in a Texas Instruments data sheet SPRS691E—November 2010—Revised March 2014 entitled: “ TMS 320 C 6678 —Multicore Fixed and Floating - Point Digital Signal Processor ”, which is incorporated in its entirety for all purposes as if fully set forth herein.

Audio signal processing typically involve analyzing, detecting, processing, simulating, or cancelling of the following affects or phenomena, or using the following techniques:

a. Echo: to simulate the effect of reverberation in a large hall or cavern, one or several delayed signals are added to the original signal. To be perceived as echo, the delay has to be of order 35 milliseconds or above. Short of actually playing a sound in the desired environment, the effect of echo can be implemented using either digital or analog methods. Analog echo effects are implemented using tape delays and/or spring reverbs. When large numbers of delayed signals are mixed over several seconds, the resulting sound has the effect of being presented in a large room, and it is more commonly called reverberation or reverb for short.

b. Flanger—to create an unusual sound, a delayed signal is added to the original signal with a continuously variable delay (usually smaller than 10 ms). This effect is now done electronically using a DSP, but originally the effect was created by playing the same recording on two synchronized tape players, and then mixing the signals together. As long as the machines were synchronized, the mix would sound more-or-less normal, but if the operator placed his finger on the flange of one of the players (hence “flanger”), that machine would slow down and its signal would fall out-of-phase with its partner, producing a phasing effect. Once the operator took his finger off, the player would speed up until its tachometer was back in phase with the master, and as this happened, the phasing effect would appear to slide up the frequency spectrum. This phasing up-and-down the register can be performed rhythmically.

c. Phaser—another way of creating an unusual sound; the signal is split, a portion is filtered with an all-pass filter to produce a phase-shift, and then the unfiltered and filtered signals are mixed. The phaser effect was originally a simpler implementation of the flanger effect since delays were difficult to implement with analog equipment. Phasers are often used to give a “synthesized” or electronic effect to natural sounds, such as human speech. The voice of C-3PO from Star Wars was created by taking the actor's voice and treating it with a phaser.

d. Chorus—a delayed signal is added to the original signal with a constant delay. The delay has to be short in order not to be perceived as echo, but above 5 ms to be audible. If the delay is too short, it will destructively interfere with the un-delayed signal and create a flanging effect. Often, the delayed signals will be slightly pitch shifted to more realistically convey the effect of multiple voices.

e. Equalization—different frequency bands are attenuated or boosted to produce desired spectral characteristics. Moderate use of equalization (often abbreviated as “EQ”) can be used to “fine-tune” the tone quality of a recording; extreme use of equalization, such as heavily cutting a certain frequency can create more effects that are unusual.

f. Filtering—Equalization is a form of filtering. In the general sense, frequency ranges can be emphasized or attenuated using low-pass, high-pass, band-pass or band-stop filters. Band-pass filtering of voice can simulate the effect of a telephone because telephones use band-pass filters. overdrive effects such as the use of a fuzz box can be used to produce distorted sounds, such as for imitating robotic voices or to simulate distorted radiotelephone traffic. The most basic overdrive effect involves clipping the signal when its absolute value exceeds a certain threshold.

g. Pitch shift—this effect shifts a signal up or down in pitch. For example, a signal may be shifted an octave up or down. This is usually applied to the entire signal and not to each note separately. Blending the original signal with shifted duplicate(s) can create harmonies from one voice. Another application of pitch shifting is pitch correction. Here a musical signal is tuned to the correct pitch using digital signal processing techniques. This effect is ubiquitous in karaoke machines and is often used to assist pop singers who sing out of tune.

h. Time stretching—the complement of pitch shift, that is, the process of changing the speed of an audio signal without affecting its pitch.

i. Resonators—emphasize harmonic frequency content on specified frequencies. These may be created from parametric EQs or from delay-based comb-filters.

j. Synthesizer—generate artificially almost any sound by either imitating natural sounds or creating completely new sounds.

k. Modulation—to change the frequency or amplitude of a carrier signal in relation to a predefined signal. Ring modulation, also known as amplitude modulation, is an effect made famous by Doctor Who's Daleks and commonly used throughout sci-fi.

l. Compression—the reduction of the dynamic range of a sound to avoid unintentional fluctuation in the dynamics. Level compression is not to be confused with audio data compression, where the amount of data is reduced without affecting the amplitude of the sound it represents.

m. Reverse echo—a swelling effect created by reversing an audio signal and recording echo and/or delay while the signal runs in reverse. When played back forward the last echoes are heard before the effected sound creating a rush like swell preceding and during playback.

n. Active noise control—a method for reducing unwanted sound.

Mel-Frequency Cepstrum (MFC). The Mel-Frequency Cepstrum (MFC) is a representation of the short-term power spectrum of a sound, based on a linear cosine transform of a log power spectrum on a nonlinear Mel scale of frequency. Mel-Frequency Cepstral Coefficients (MFCCs) are coefficients that collectively make up an MFC. The difference between the Cepstrum and the Mel-frequency Cepstrum is that in the MFC, the frequency bands are equally spaced on the Mel scale, which approximates the human auditory system's response more closely than the linearly-spaced frequency bands used in the normal Cepstrum. This frequency warping can allow for better representation of sound, for example, in audio compression.

MFCCs are commonly derived by the steps of: taking the Fourier transform of (a windowed excerpt of) a signal, mapping the powers of the spectrum obtained above onto the Mel scale, using triangular overlapping windows; taking the logs of the powers at each of the Mel frequencies; taking the discrete cosine transform of the list of Mel log powers, as if it were a signal, and the MFCCs are the amplitudes of the resulting spectrum. There can be variations on this process, for example: differences in the shape or spacing of the windows used to map the scale, or addition of dynamics features such as “delta” and “delta-delta” (first- and second-order frame-to-frame difference) coefficients. Calculating and using MFCC is further described in European Telecommunications Standards Institute (ETSI) 2003 Standard ETSI ES 201 108 v1.1.3 (2003-09) entitled: “ Speech Processing, Transmission and quality Aspects ( STQ ); Distributed speech recognition; Front - end feature extraction algorithm; Compression algorithms ”, in an article in J. Computer Science & Technology, 16(6):582-589, September 2001 by Fang Zheng, Guoliang Zhang, and Zhanjiang Song entitled: “ Comparison of Different Implementations of MFCC ”, and in RWTH Aachen, University of Technology, Aachen Germany publication by Sirko Molau, Michael Pitz, Ralf Schluter, and Hermann Ney, entitled: “ Computing MEL - Frequency Cepstral Coefficients on the Power Spectrum ”, which are all incorporated in their entirety for all purposes as if fully set forth herein.

LPC. Linear Predictive Coding (LPC) is a tool used mostly in audio signal processing and speech processing for representing the spectral envelope of a digital signal of speech in compressed form, using the information of a linear predictive model. It is one of the most powerful speech analysis techniques, and one of the most useful methods for encoding good quality speech at a low bit rate and provides extremely accurate estimates of speech parameters. LPC is described in a Technion-Haifa, Israel presentation by Nimrod Peleg (Updated March 2009) entitled: “ Linear Prediction Coding ”, and in a book by P. P. Vaidyanathan entitled: “ The Theory of Linear Prediction ” published 2008 [ISBN: 1598295756], which are both incorporated in their entirety for all purposes as if fully set forth herein.

LPC starts with the assumption that a speech signal is produced by a buzzer at the end of a tube (voiced sounds), with occasional added hissing and popping sounds (sibilants and plosive sounds). Although apparently crude, this model is actually a close approximation of the reality of speech production. The glottis (the space between the vocal folds) produces the buzz, which is characterized by its intensity (loudness) and frequency (pitch). The vocal tract (the throat and mouth) forms the tube, which is characterized by its resonances, which give rise to formants, or enhanced frequency bands in the sound produced. Hisses and pops are generated by the action of the tongue, lips and throat during sibilants and plosives. LPC analyzes the speech signal by estimating the formants, removing their effects from the speech signal, and estimating the intensity and frequency of the remaining buzz. The process of removing the formants is called inverse filtering, and the remaining signal after the subtraction of the filtered modeled signal is called the residue.

The numbers that describe the intensity and frequency of the buzz, the formants, and the residue signal, can be stored or transmitted somewhere else. LPC synthesizes the speech signal by reversing the process: use the buzz parameters and the residue to create a source signal, use the formants to create a filter (which represents the tube), and run the source through the filter, resulting in speech. Because speech signals vary with time, this process is done on short chunks of the speech signal, which are called frames, generally 30 to 50 frames per second give intelligible speech with good compression. LPC is frequently used for transmitting spectral envelope information, and as thus has to be tolerant of transmission errors. Transmission of the filter coefficients directly is undesirable, since they are very sensitive to errors. In other words, a very small error can distort the whole spectrum, or worse, a small error might make the prediction filter unstable. There are more advanced representations such as Log Area Ratios (LAR), Line Spectral Pairs (LSP) decomposition and reflection coefficients. Of these, especially LSP decomposition has gained popularity, since it ensures stability of the predictor, and spectral errors are local for small coefficient deviations.

Time—frequency Analysis. A time-frequency analysis comprises those techniques that study a signal in both the time and frequency domains simultaneously, using various time-frequency representations. Rather than viewing a 1-dimensional signal (a function, real or complex-valued, whose domain is the real line) and some transform (another function whose domain is the real line, obtained from the original via some transform), time-frequency analysis studies a two-dimensional signal—a function whose domain is the two-dimensional real plane, obtained from the signal via a time-frequency transform. Time-Frequency analysis is described in an article by Rolf Hut (September 2004) entitled: “ Time Frequency Analysis—a Comparison between cochlear modeling and existing methods ”, and in an article by Franz Hlawatsch and Gerald Matz (of the Institute of Communications and radio-Frequency Engineering, Vienna University of Technology) entitled: “ Time - Frequency Signal Processing: A Statistical Perspective ”, which are both incorporated in their entirety for all purposes as if fully set forth herein. One of the most basic forms of time-frequency analysis is the Short-Time Fourier Transform (STFT), but techniques that are more sophisticated have been developed, such as wavelets.

There are several different ways to formulate a valid time-frequency distribution function, resulting in several well-known time-frequency distributions, such as: Short-time Fourier transform (including the Gabor transform); Wavelet transform; Bilinear time-frequency distribution function (Wigner distribution function, or WDF); and Modified Wigner distribution function or Gabor-Wigner distribution function.

To analyze the signals well, choosing an appropriate time-frequency distribution function is important. Which time-frequency distribution function should be used depends on the application being considered, as shown by reviewing a list of applications. The high clarity of the Wigner Distribution Function (WDF) obtained for some signals is due to the auto-correlation function inherent in its formulation; however, the latter also causes the cross-term problem. Therefore, if we want to analyze a single-term signal, using the WDF may be the best approach; if the signal is composed of multiple components, some other methods like the Gabor transform, Gabor-Wigner distribution or Modified B-Distribution functions may be better choices.

HMD. A Head-Mounted Display (or Helmet-Mounted Display, for aviation applications), both abbreviated HMD, is a display device, worn on the head or as part of a helmet, that has a small display optic in front of one (monocular HMD) or each eye (binocular HMD). There is also an Optical head-mounted display (OHMD), which is a wearable display that has the capability of reflecting projected images as well as allowing the user to see through it. A typical HMD has either one or two small displays with lenses and semi-transparent mirrors embedded in a helmet, eyeglasses (also known as data glasses) or visor. The display units are miniaturized and may include CRT, LCDs, Liquid crystal on silicon (LCos), or OLED. Some vendors employ multiple micro-displays to increase total resolution and field of view. An HMD 47 b is pictorially depicted in FIG. 4 b , and includes a horizontal strap 48 a and a vertical strap 48 b for head wearing by a person. A wireless-capable HMD 47 a is pictorially depicted in FIG. 4 b , shown to include an antenna 49 a and an antenna 49 b for wireless communication. The wireless-capable HMD 47 a is shown worn by a person 36 in a view 47 c shown in FIG. 4 c.

HMDs differ in whether they can display just a Computer Generated Image (CGI), show live images from the real world or a combination of both. Most HMDs display only a computer-generated image, sometimes referred to as a virtual image. Some HMDs allow a CGI to be superimposed on a real-world view. This is sometimes referred to as augmented reality or mixed reality. Combining real-world view with CGI can be done by projecting the CGI through a partially reflective mirror and viewing the real world directly. This method is often called Optical See-Through. Combining real-world view with CGI can also be done electronically by accepting video from a camera and mixing it electronically with CGI. This method is often called Video See-Through.

An optical head-mounted display uses an optical mixer, which is made of partly silvered mirrors. It has the capability of reflecting artificial images as well as letting real images to cross the lens and let the user to look through it. Various techniques have existed for see-through HMD's. Most of these techniques can be summarized into two main families: “Curved Mirror” based and “Waveguide” based. Various waveguide techniques have existed for some time. These techniques include diffraction optics, holographic optics, polarized optics, and reflective optics. Major HMD applications include military, governmental (fire, police, etc.) and civilian/commercial (medicine, video gaming, sports, etc.).

The Virtual Reality (VR) technology most fundamental to the proposed research is the Head-Mounted Display (HMD). An HMD is a helmet or visor worn by the user with two screens, one for each eye, so that a stereoscopic “true 3D” image may be displayed to the user. This is achieved by displaying the same image in each screen, but offset by a distance equal to the distance between the user's eyes, mimicking how human vision perceives the world. HMDs can be opaque or see-through. In a see-through HMD, the screens are transparent so that the user can see the real world as well as what is being displayed on the screens. However, see-through HMDs often suffer from brightness problems that make them difficult to use in variable lighting conditions. Most opaque HMD designs block out the real world so that the user can only see the screens, thereby providing an immersive experience.

Some HMDs are used in conjunction with tracking systems. By tracking the user's position or orientation (or both), the system can allow the user to move naturally via locomotion and by turning their head and body, and update the graphical display accordingly. This allows for natural exploration of virtual environments without needing to rely on a keyboard, mouse, joystick, and similar interface hardware. Positional tracking is often accomplished by attaching markers (such as infrared markers) to the HMD or the user's body and using multiple special cameras to track the location of these markers in 3D space. Orientation tracking can be accomplished using an inertial tracker, which uses a sensor to detect velocities on three axes. Some systems use a combination of optical and inertial tracking, and other tracking techniques (e.g., magnetic) also exist. The output from the tracking systems is fed into the computer rendering the graphical display so that it can update the scene. Filtering is usually necessary to make the data usable since it comes in the form of noisy analog measurements. An HMD typically includes a horizontal strap and a vertical strap for head wearing by a person. A wireless-capable HMD typically includes an antenna for wireless communication.

Methods and systems for capturing an image are provided in U.S. Patent Application Publication No. 2013/0222638 to Wheeler et al. entitled: “Image Capture Based on Gaze Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. In one example, a head-mounted device (HMD) having an image capturing device, a viewfinder, a gaze acquisition system, and a controller may be configured to capture an image. The image capturing device may be configured to have an imaging field of view including at least a portion of a field of view provided by the viewfinder. The gaze acquisition system may be configured to acquire a gaze direction of a wearer. The controller may be configured to determine whether the acquired gaze direction is through the viewfinder and generate an image capture instruction based on a determination that the acquired gaze direction indicates a gaze through the viewfinder. The controller may further be configured to cause the image capturing device to capture an image.

Methods and systems for capturing and storing an image are provided in U.S. Pat. No. 8,941,561 to Starner entitled: “Image Capture”, which is incorporated in its entirety for all purposes as if fully set forth herein. In one example, eye-movement data associated with a head-mountable device (HMD) may be received. The HMD may include an image-capture device arranged to capture image data corresponding to a wearer-view associated with the HMD. In one case, the received eye-movement data may indicate sustained gaze. In this case, a location of the sustained gaze may be determined, and an image including a view of the location of the sustained gaze may be captured. At least one indication of a context of the captured image, such as time and/or geographic location of the HMD when the image was captured may be determined and stored in a data-item attribute database as part of a record of the captured image. In a further example, movements associated with the HMD may also be determined and based on to determine sustained gaze and the location of the sustained gaze.

A head mountable display (HMD) system is disclosed in U.S. Patent Application Publication No. 2014/0362446 to Bickerstaff et al. entitled: “Electronic Correction Based on Eye Tracking”, which is incorporated in its entirety for all purposes as if fully set forth herein. The head mountable display (HMD) system comprises an eye position detector comprising one or more cameras configured to detect the position of each of the HMD user's eyes; a dominant eye detector configured to detect a dominant eye of the HMD user; and an image generator configured to generate images for display by the HMD in dependence upon the HMD user's eye positions, the image generator being configured to apply a greater weight to the detected position of the dominant eye than to the detected position of the non-dominant eye.

Methods and systems are described that involve a headmountable display (HMD) or an associated device determining the orientation of a person's head relative to their body, are described in U.S. Pat. No. 9,268,136 to Patrick et al. entitled: “Use of Comparative Sensor Data to Determine Orientation of Head Relative to Body”, which is incorporated in its entirety for all purposes as if fully set forth herein. To do so, example methods and systems may compare sensor data from the HMD to corresponding sensor data from a tracking device that is expected to move in a manner that follows the wearer's body, such a mobile phone that is located in the HMD wearer's pocket.

A Head Mountable Display (HMD) system in which images are generated for display to the user is described in Patent Cooperation Treaty (PCT) International Application (IA) Publication No. WO 2014/199155 to Ashforth et al. entitled: “Head-Mountable Apparatus and Systems”, which is incorporated in its entirety for all purposes as if fully set forth herein. The head mountable display (HMD) system comprises a detector configured to detect the eye position and/or orientation and/or the head orientation of the HMD wearer, and a controller configured to control the generation of images for display, at least in part, according to the detection of the eye position and/or orientation and/or the head orientation of the HMD wearer; in which the controller is configured to change the display of one or more image features according to whether or not the user is currently looking at those image features, the image features are menu items or information items, by rendering an image feature so as to be more prominent on the display if the user is looking at it, such that the image feature is enlarged, moved from a peripheral display position, replaced by a larger image feature and/or brought forward in a 3D display space if the user is looking at it.

AR. Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. AR can be defined as a system that fulfills three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. The overlaid sensory information can be constructive (i.e., additive to the natural environment), or destructive (i.e. masking of the natural environment). This experience is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment. In this way, augmented reality alters one's ongoing perception of a real-world environment, whereas virtual reality completely replaces the user's real-world environment with a simulated one. Augmented reality is related to two largely synonymous terms: mixed reality and computer-mediated reality.

The primary value of augmented reality is the manner in which components of the digital world blend into a person's perception of the real world, not as a simple display of data, but through the integration of immersive sensations, which are perceived as natural parts of an environment. Augmented reality is used to enhance natural environments or situations and offer perceptually enriched experiences. With the help of advanced AR technologies (e.g. adding computer vision, incorporating AR cameras into smartphone applications and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulated. Information about the environment and its objects is overlaid on the real world. This information can be virtual or real, such as seeing other real sensed or measured information such as electromagnetic radio waves overlaid in exact alignment with where they actually are in space. Augmented reality also has a lot of potential in the gathering and sharing of tacit knowledge. Augmentation techniques are typically performed in real time and in semantic contexts with environmental elements. Immersive perceptual information is sometimes combined with supplemental information like scores over a live video feed of a sporting event. This combines the benefits of both augmented reality technology and heads up display technology (HUD).

Typical hardware components for augmented reality are: a processor, display, sensors and input devices. Modern mobile computing devices like smartphones and tablet computers contain these elements, which often include a camera and microelectromechanical systems (MEMS) sensors such as an accelerometer, GPS, and solid state compass, making them suitable AR platforms. There are two technologies used in augmented reality: diffractive waveguides and reflective waveguides.

Various technologies are used in augmented reality rendering, including optical projection systems, monitors, handheld devices, and display systems, which are worn on the human body. A Head-Mounted Display (HMD) is a display device worn on the forehead, such as a harness or helmet-mounted. HMDs place images of both the physical world and virtual objects over the user's field of view. Modern HMDs often employ sensors for six degrees of freedom monitoring that allow the system to align virtual information to the physical world and adjust accordingly with the user's head movements. HMDs can provide VR users with mobile and collaborative experiences.

AR displays can be rendered on devices resembling eyeglasses. Versions include eyewear that employs cameras to intercept the real world view and re-display its augmented view through the eyepieces and devices in which the AR imagery is projected through or reflected off the surfaces of the eyewear lens pieces.

HUD. A Head-Up Display (HUD) is a transparent display that presents data without requiring users to look away from their usual viewpoints. Near-eye augmented reality devices can be used as portable head-up displays as they can show data, information, and images while the user views the real world. Many definitions of augmented reality only define it as overlaying the information. This is basically what a head-up display does; however, practically speaking, augmented reality is expected to include registration and tracking between the superimposed perceptions, sensations, information, data, and images and some portion of the real world.

Contact lenses. Contact lenses that display AR imaging are in development. These bionic contact lenses might contain the elements for display embedded into the lens including integrated circuitry, LEDs and an antenna for wireless communication.

Virtual retinal display. A Virtual Retinal Display (VRD) is a personal display device where a display is scanned directly onto the retina of a viewer's eye. This results in bright images with high resolution and high contrast, and the viewer sees what appears to be a conventional display floating in space. Virtual retinal display creates images that can be seen in ambient daylight and ambient room light. The VRD is considered a preferred candidate to use in a surgical display due to its combination of high resolution and high contrast and brightness. Additional tests show high potential for VRD to be used as a display technology for patients that have low vision.

Handheld. A Handheld display employs a small display that fits in a user's hand. All handheld AR solutions to date opt for video see-through. Initially handheld AR employed fiducial markers, and later GPS units and MEMS sensors such as digital compasses and six degrees of freedom accelerometer-gyroscope. Today Simultaneous Localization and Mapping (SLAM) markerless trackers such as PTAM (Parallel Tracking and Mapping) are starting to come into use. Handheld display AR promises to be the first commercial success for AR technologies. The two main advantages of handheld AR are the portable nature of handheld devices and the ubiquitous nature of camera phones. The disadvantages are the physical constraints of the user having to hold the handheld device out in front of them at all times, as well as the distorting effect of classically wide-angled mobile phone cameras when compared to the real world as viewed through the eye.

Spatial Augmented Reality (SAR) augments real-world objects and scenes, without the use of special displays such as monitors, head-mounted displays or hand-held devices. SAR makes use of digital projectors to display graphical information onto physical objects. The key difference in SAR is that the display is separated from the users of the system. Since the displays are not associated with each user, SAR scales naturally up to groups of users, allowing for collocated collaboration between users.

Other applications include table and wall projections. One innovation, the Extended Virtual Table, separates the virtual from the real by including beam-splitter mirrors attached to the ceiling at an adjustable angle. Virtual showcases, which employ beam splitter mirrors together with multiple graphics displays, provide an interactive means of simultaneously engaging with the virtual and the real. Many more implementations and configurations make spatial augmented reality display an increasingly attractive interactive alternative. A SAR system can display on any number of surfaces in an indoor setting at once. SAR supports both a graphical visualization and passive haptic sensation for the end users. Users are able to touch physical objects in a process that provides passive haptic sensation.

A virtual reality composer platform and system (VRCPS) is described in U.S. Pat. No. 7,754,955 to Egan entitled: “Virtual reality composer platform system”, which is incorporated in its entirety for all purposes as if fully set forth herein. The system includes a plurality of user input/output devices and signal input/output controllers interfaced to a central processing unit complete with plurality of memory means, a butterfly morpheus musical instrument with plurality of manual input means each with a first unique visible indicia interfaced to said central processing unit, a plurality of finger adapters each with a second unique visible indicia donned on respective fingers and at least one custom butterfly Morpheus music notation computer interface. The system is particularly suited for composing music for self-learning and teaching for all types of musical instruments for optimal virtual reality multimedia experience. The VRCPS platform and concepts disclosed are vari-dimensional acoustic environments, which are equally suited to all types of electronic learning and composing systems, game systems and computers. It is suitable for all levels of Do-It-Yourself learning from learning Beginners to Virtuoso Levels.

A method, apparatus, and User Interface, and product for assisting users learning to play the Chords of any selected Song are described in U.S. Pat. No. 10,614,786 to Barry entitled: “Musical chord identification, selection and playing method and means for physical and virtual musical instruments”, which is incorporated in its entirety for all purposes as if fully set forth herein. The method, apparatus, and User Interface, and product quickly and easily provide means to quickly and easily generate the individual Note sounds for the Chords of the selected Song employing a broad range of Virtual and Physical Instrument.

An augmented reality based piano performance assistant method which enables a user who is not familiar with a sheet music to play a piano, and a device performing the same is provided in South-Korea Patent Application Publication KR20170138135 entitled: “Method of helping piano performance and based on augemented reality and apparatus performing the same”, which is incorporated in its entirety for all purposes as if fully set forth herein. According to an embodiment of the present invention, the augmented reality based piano performance assistant method which is performed by an augmented reality based piano performance assistant device comprises the following steps of: recognizing a plurality of octave recognition labels located on a keyboard of the piano device; loading a virtual keyboard with respect to keyboard information in an augmented reality environment by using the information on the octave corresponding to the plurality of octave recognition labels; and displaying piano performance assistant information on the virtual keyboard loaded in the augmented reality environment according to performance information received from a performance information providing server when performance is started.

A kind of piano training system and method based on mixed reality is described in China Patent Application Publication CN109493686 entitled: “A kind of piano training system and method based on mixed reality”, which is incorporated in its entirety for all purposes as if fully set forth herein. The piano training system includes the mixed reality helmet, piano, positioning device and processing unit; the piano is true. The positioning device is arranged on the piano or the mixed reality helmet, for relative position between the key and the mixed reality helmet of the real-time measurement piano. The processing unit, signal connect the mixed reality helmet, for loading and according to the corresponding virtual training scene of standard piano file generated, and by the virtual training scene transfer to the mixed reality helmet. The mixed reality helmet, signal connects the positioning device, for user's body-worn, the virtual training scene is shown in the upside of the key of the piano for relative position between the key and the key and the mixed reality helmet of the piano that detect according to the positioning device of the real piano. Its enjoyment that can increase piano training process and permission user adjust the sitting posture of oneself in the training process to prevent cervical vertebra over fatigue.

A computer implemented method for providing an augmented reality (AR) function is described in U.S. Pat. No. 10,482,862 to Hämäläinen et al. entitled: “Computer implemented method for providing augmented reality (AR) function regarding music track”, which is incorporated in its entirety for all purposes as if fully set forth herein. The method comprises receiving input information regarding a music track and an instrument; determining attribute information of the music track based on the received input information; receiving real time content of audiovisual (AV) input signals using at least one capturing device; generating visual information corresponding to visual data of the real time content, wherein the visual information corresponds to a view regarding at least one user limb and an instrument comprising a plurality of user operable elements; generating augmented reality (AR) instruction information based on the attribute information of the music track, the augmented reality (AR) instruction information comprising a plurality of layers; and generating augmented reality (AR) visual information by applying the augmented reality (AR) instruction information to the visual information so that a first layer of the augmented reality (AR) instruction information is applied above at least a portion of the visual information.

A system enabling the performance of sensory stimulating content including music and video using gaming in a cyber reality environment, such as using a virtual reality headset, is described in U.S. Pat. No. 10,418,008 to Bencar et al. entitled: “Cyber reality device including gaming based on a plurality of musical programs”, which is incorporated in its entirety for all purposes as if fully set forth herein. This disclosure includes a system and method through which a performer can virtually trigger and control a presentation of pre-packaged sensory stimulating content including musical programs through gaming. A theme for the performer is that the pre-packaged sensory stimulating content is preferably chosen such that, even where the performer is a novice, the sensory stimulating data is presented in a pleasing and sympathetic manner and scoring is provided as a function of the performer's ability to provide a gesture in association with a displayed virtual trigger.

Musical notation. Music notation (or musical notation) is any system, convention, or standard used to visually represent aurally perceived music played with musical instruments or sung by the human voice through the use of written, printed, or otherwise-produced symbols, including notation for durations of absence of sound such as rests. Typically, in this framework pitches are indicated by placing oval noteheads on the staff lines or between the lines, and the pitch of the oval musical noteheads can be modified by accidentals. The duration (note length) is shown with different note values, which can be indicated by the notehead being a stemless hollow oval (a whole note or semibreve), a hollow rectangle or stemless hollow oval with one or two vertical lines on either side (double whole note or breve), a stemmed hollow oval (a half note or minim), or solid oval using stems to indicate quarter notes (crotchets) and stems with added flags or beams to indicate smaller subdivisions, and additional symbols such as dots and ties which lengthen the duration of a note. Notation is read from left to right, which makes setting music for right-to-left scripts difficult.

A staff (or stave, in British English) of written music generally begins with a clef, which indicates the position of one particular note on the staff. The treble clef or G clef was originally a letter G and it identifies the second line up on the five line staff as the note G above middle C. The bass clef or F clef shows the position of the note F below middle C. While the treble and bass clef are the most widely used clefs, other clefs are used, such as the alto clef (used for viola and alto trombone music) and the tenor clef (used for some cello, tenor trombone, and double bass music). Notes representing a pitch outside of the scope of the five line staff can be represented using ledger lines, which provide a single note with additional lines and spaces. Some instruments use mainly one clef, such as violin and flute, which use treble clef and double bass and tuba, which use bass clef. Some instruments regularly use both clefs, such as piano and pipe organ.

Following the clef, the key signature on a staff indicates the key of the piece or song by specifying that certain notes are flat or sharp throughout the piece, unless otherwise indicated with accidentals added before certain notes. When a sharp is placed before a note, this makes that note one semitone higher. When a flat is placed before a note, this makes that note one semitone lower. Double sharps and double flats are less common, but they are used. A double sharp is placed before a note to make it two semitones higher. A double flat is placed before a note to make it two semitones lower. A natural sign placed before a note renders that note in its “natural” form, which means that any sharps or flats applying to that note from the key signature or from accidentals are cancelled. Sometimes a courtesy accidental is used in music where it is not technically required, to remind the musician of what pitch the key signature requires. Following the key signature is the time signature. The time signature typically consists of two numbers, with one of the most common being The top “4” indicates that there are four beats per measure (also called bar). The bottom “4” indicates that each of those beats are quarter notes. Measures divide the piece into groups of beats, and the time signatures specify those groupings.

Musical symbols. Musical symbols are marks and symbols used in musical notation of musical scores. Some are used to notate pitch, tempo, metre, duration, and articulation of a note or a passage of music. In some cases, symbols provide information about the form of a piece (e.g., how many repeats of a section) or about how to play the note (e.g., with violin family instruments, a note may be bowed or plucked). Some symbols are instrument-specific notation giving the performer information about which finger, hand or foot to use. Selected examples of popular musical symbols according to a popular music notation convention are described in a view 60 shown in FIG. 6 , and a correspondence of musical symbols to the associated piano keys is described in a view 60 a shown in FIG. 6 a.

Clefs define the pitch range, or tessitura, of the staff on which it is placed. A clef is usually the leftmost symbol on a staff. Additional clefs may appear in the middle of a staff to indicate a change in register for instruments with a wide range. In early music, clefs could be placed on any of several lines on a staff. Musical note and rest values are not absolutely defined, but are proportional in duration to all other note and rest values. The whole note is the reference value, and the other notes are named (in American usage) in comparison; i.e., a quarter note is a quarter of the length of a whole note. Accidentals modify the pitch of the notes that follow them on the same staff position within a measure, unless cancelled by an additional accidental. Key signatures define the prevailing key of the music that follows, thus avoiding the use of accidentals for many notes. If no key signature appears, the key is assumed to be C major/A minor, but can also signify a neutral key, employing individual accidentals as required for each note. The key signature examples shown here are described as they would appear on a treble staff.

Time signatures define the meter of the music. Music is “marked off” in uniform sections called bars or measures, and time signatures establish the number of beats in each. This does not necessarily indicate which beats to emphasize, however, so a time signature that conveys information about the way the piece actually sounds is thus chosen. Time signatures tend to suggest prevailing groupings of beats or pulses. Articulations (or accents) specify how to perform individual notes within a phrase or passage. They can be fine-tuned by combining more than one such symbol over or under a note. They may also appear in conjunction with phrasing marks listed above.

Sheet music. Sheet music is a handwritten or printed form of musical notation that uses musical symbols to indicate the pitches, rhythms, or chords of a song or instrumental musical piece. The medium of sheet music traditionally was a paper, however modern mediums include the presentation of musical notation on computer screens and the development of scorewriter computer programs that can notate a song or piece electronically, and, in some cases, “play back” the notated music using a synthesizer or virtual instruments.

Sheet music can be used as a record of, a guide to, or a means to perform, a song or a musical piece. Sheet music enables instrumental performers who are able to read music notation (a pianist, orchestral instrument players, a jazz band, etc.) or singers to perform a song or piece. The intended purpose of an edition of sheet music affects its design and layout. If sheet music is intended for study purposes, as in a music history class, the notes and staff can be made smaller and the editor does not have to be worried about page turns. In classical music, authoritative musical information about a piece can be gained by studying the written sketches and early versions of compositions that the composer might have retained, as well as the final autograph score and personal markings on proofs and printed scores. An example of a sheet music of a popular song is described in a view 60 b shown in FIG. 6 b.

Musical instrument. A musical instrument is a device created or adapted to make musical sounds. There are many different methods of classifying musical instruments. Various methods examine aspects such as the physical properties of the instrument (material, color, shape, etc.), the use for the instrument, the means by which music is produced with the instrument, the range of the instrument, and the instrument's place in an orchestra or other ensemble. Most methods are specific to a geographic area or cultural group and were developed to serve the unique classification requirements of the group.

The most commonly used system divides instruments into string instruments, woodwind instruments, brass instruments and percussion instruments. Musical instruments are also often classified by their musical range in comparison with other instruments in the same family. This may be useful when placing instruments in context of an orchestra or other ensemble. These terms are named after singing voice classifications, and include Soprano instruments, such as flute, violin, soprano saxophone, trumpet, clarinet, oboe, and piccolo; Alto instruments, such as alto saxophone, French horn, English horn, viola, and alto horn; Tenor instruments, such as trombone, tenoroon, tenor saxophone, tenor violin, guitar, and tenor drum; Baritone instruments, such as bassoon, baritone saxophone, bass clarinet, cello, baritone horn, and euphonium; and Bass instruments, such as double bass, bass guitar, contrabassoon, bass saxophone, tuba, and bass drum.

Some instruments fall into more than one category. For example, the cello may be considered tenor, baritone or bass, depending on how its music fits into the ensemble. The trombone and French horn may be alto, tenor, baritone, or bass depending on the range it is played in. Many instruments have their range as part of their name: soprano saxophone, tenor saxophone, baritone horn, alto flute, bass guitar, etc. Additional adjectives describe instruments above the soprano range or below the bass, for example the sopranino saxophone and contrabass clarinet. When used in the name of an instrument, these terms are relative, describing the instrument's range in comparison to other instruments of its family and not in comparison to the human voice range or instruments of other families.

The original Hornbostel-Sachs system classified instruments into four main groups: Idiophones, which produce sound by vibrating the primary body of the instrument itself; they are sorted into concussion, percussion, shaken, scraped, split, and plucked idiophones, such as claves, xylophone guiro, slit drum, mbira, and rattle; Membranophones, which produce sound by a vibrating a stretched membrane; they may be drums (further sorted by the shape of the shell), which are struck by hand, with a stick, or rubbed, but kazoos and other instruments that use a stretched membrane for the primary sound (not simply to modify sound produced in another way) are also considered membranophones; Chordophones, which produce sound by vibrating one or more strings; they are sorted into according to the relationship between the string(s) and the sounding board or chamber (for example, if the strings are laid out parallel to the sounding board and there is no neck, the instrument is a zither whether it is plucked like an autoharp or struck with hammers like a piano. If the instrument has strings parallel to the sounding board or chamber and the strings extend past the board with a neck, then the instrument is a lute, whether the sound chamber is constructed of wood like a guitar or uses a membrane like a banjo); Aerophones, which produce a sound with a vibrating column of air; they are sorted into free aerophones such as a bullroarer or whip, which move freely through the air; reedless aerophones, such as flutes and recorders, which cause the air to pass over a sharp edge; reed instruments, which use a vibrating reed (this category may be further divided into two classifications: single-reeded and double-reeded instruments. Examples of the former are clarinets and saxophones, while the latter includes oboes and bassoons); and lip-vibrated aerophones such as trumpets, trombones and tubas, for which the lips themselves function as vibrating reeds.

String instruments. String instruments, also known as stringed instruments, or chordophones, are musical instruments that produce sound from vibrating strings when the performer plays or sounds the strings in some manner. Musicians play some string instruments by plucking the strings with their fingers or a plectrum—and others by hitting the strings with a light wooden hammer or by rubbing the strings with a bow. In some keyboard instruments, such as the harpsichord, the musician presses a key that plucks the string. With bowed instruments, the player pulls a rosined horsehair bow across the strings, causing them to vibrate. With a hurdy-gurdy, the musician cranks a wheel whose rosined edge touches the strings. In most string instruments, the vibrations are transmitted to the body of the instrument, which often incorporates some sort of hollow or enclosed area. The body of the instrument also vibrates, along with the air inside it. The vibration of the body of the instrument and the enclosed hollow or chamber make the vibration of the string more audible to the performer and audience. The body of most string instruments is hollow. Some, however—such as electric guitar and other instruments that rely on electronic amplification—may have a solid wood body. The most common string instruments in the string family are guitar, electric bass, violin, viola, cello, double bass, banjo, mandolin, ukulele, and harp. Pictorial views of a violin 51 , a guitar 51 a , a mandolin 51 b , a banjo 51 c , and a harp 51 d , are shown in FIG. 5 .

Bowed instruments include the string section instruments of the Classical music orchestra (violin, viola, cello and double bass) and a number of other instruments (e.g., viols and gambas used in early music from the Baroque music era and fiddles used in many types of folk music). All of the bowed string instruments can also be plucked with the fingers, a technique called “pizzicato”. A wide variety of techniques are used to sound notes on the electric guitar, including plucking with the fingernails or a plectrum, strumming and even “tapping” on the fingerboard and using feedback from a loud, distorted guitar amplifier to produce a sustained sound. Some types of string instrument are mainly plucked, such as the harp and the electric bass. Other examples include the sitar, rebab, banjo, mandolin, ukulele, and bouzouki.

String instruments can be divided in three groups: Lutes, which are instruments that support the strings via a neck and a bout (“gourd”), for instance a guitar, a violin, or a saz, Harps, which are instruments that contain the strings within a frame, and Zithers, which are instruments that have the strings mounted on a body, frame or tube, such as a guqin, a cimbalom, an autoharp, harpsichord, a piano, or a valiha.

It is also possible to divide the instruments into categories focused on how the instrument is played. An acoustic guitar being strummed. All string instruments produce sound from one or more vibrating strings, transferred to the air by the body of the instrument (or by a pickup in the case of electronically amplified instruments). They are usually categorised by the technique used to make the strings vibrate (or by the primary technique, in the case of instruments where more than one may apply). The three most common techniques are plucking, bowing, and striking. An important difference between bowing and plucking is that in the former the phenomenon is periodic so that the overtones are kept in a strictly harmonic relationship to the fundamental.

Plucking is a method of playing on instruments such as the veena, banjo, ukulele, guitar, harp, lute, mandolin, oud, and sitar, using either a finger, thumb, or quills (now plastic plectra) to pluck the strings. Instruments normally played by bowing may also be plucked, a technique referred to by the Italian term pizzicato.

Bowing is a method used in some string instruments, including the violin, viola, cello, and the double bass (of the violin family), and the old viol family. The bow consists of a stick with a “ribbon” of parallel horse tail hairs stretched between its ends. The hair is coated with rosin so it can grip the string; moving the hair across a string causes a stick-slip phenomenon, making the string vibrate, and prompting the instrument to emit sound. Darker grades of rosin grip well in cool, dry climates, but may be too sticky in warmer, more humid weather. Violin and viola players generally use harder, lighter-colored rosin than players of lower-pitched instruments, who tend to favor darker, softer rosin.

The third common method of sound production in stringed instruments is to strike the string. The piano and hammered dulcimer use this method of sound production. Even though the piano strikes the strings, the use of felt hammers means that the sound that is produced can nevertheless be mellow and rounded, in contrast to the sharp attack produced when a very hard hammer strikes the strings. Violin family string instrument players are occasionally instructed to strike the string with the stick of the bow, a technique called col legno. This yields a percussive sound along with the pitch of the note.

Acoustic instruments are based on a vibrating string strung on a very thick log, as a hypothetical example, would make only a very quiet sound, so string instruments are usually constructed in such a way that the vibrating string is coupled to a hollow resonating chamber, a soundboard, or both. On the violin, for example, the four strings pass over a thin wooden bridge resting on a hollow box (the body of the violin). The normal force applied to the body from the strings is supported in part by a small cylinder of wood called the soundpost. The violin body also has two “f-holes” carved on the top. The strings' vibrations are distributed via the bridge and soundpost to all surfaces of the instrument, and are thus made louder by matching of the acoustic impedance. The correct technical explanation is that they allow a better match to the acoustic impedance of the air.

All lute type instruments traditionally have a bridge, which holds the string at the proper action height from the fret/finger board at one end of the strings. On acoustic instruments, the bridge performs an equally important function of transmitting string energy into the “sound box” of the instrument, thereby increasing the sound volume. The specific design, and materials the used in the construction of the bridge of an instrument, have a dramatic impact upon both the sound and responsiveness of the instrument. Acoustic instruments can also be made out of artificial materials, such as carbon fiber and fiberglass (particularly the larger, lower-pitched instruments, such as cellos and basses).

String instruments may be based on electronic amplification, where they are fitted with piezoelectric or magnetic pickups to convert the string's vibrations into an electrical signal that is amplified and then converted back into sound by loudspeakers. Some players attach a pickup to their traditional string instrument to “electrify” it. Another option is to use a solid-bodied instrument, which reduces unwanted feedback howls or squeals. Amplified string instruments can be much louder than their acoustic counterparts, so musicians can play them in relatively loud rock, blues, and jazz ensembles. Amplified instruments can also have their amplified tone modified by using electronic effects such as distortion, reverb, or wah-wah. Bass-register string instruments such as the double bass and the electric bass are amplified with bass instrument amplifiers that are designed to reproduce low-frequency sounds. To modify the tone of amplified bass instruments, a range of electronic bass effects are available, such as distortion and chorus.

Woodwind instrument. A woodwind instrument is a musical instrument which produces sound when the player blows air against a sharp edge or through a reed, causing the air within its resonator (usually a column of air) to vibrate. Most of these instruments are made of wood but can be made of other materials, such as metals or plastics. Woodwind instruments are a family of musical instruments within the more general category of wind instruments. Common examples include flute, clarinet, oboe, saxophone, and bassoon. There are two main types of woodwind instruments: flutes and reed instruments (otherwise called reed pipes). The main distinction between these instruments and other wind instruments is the way in which they produce sound. All woodwinds produce sound by splitting the air blown into them on a sharp edge, such as a reed or a fipple. Despite the name, a woodwind may be made of any material, not just wood, and common examples include brass, silver, cane, as well as other metals such as gold and platinum. The saxophone, for example, though made of brass, is considered a woodwind because it requires a reed to produce sound.

The modern orchestra's woodwind section typically includes: flutes, oboes, clarinets, and bassoons. The piccolo, cor anglais, bass clarinet, E-flat clarinet, and contrabassoon are commonly used supplementary woodwind instruments. The section may also on occasion be expanded by the addition of saxophone(s). The concert band's woodwind section typically includes piccolos, flutes, oboes, bass clarinets, bassoons, alto saxophones, tenor saxophones, and baritone saxophones.

Pictorial views of a Flute 52 , a Piccolo 52 a , a Clarinet 52 b , a Bass Clarinet 52 c , a Bassoon 52 d , a Contra Bassoon 52 e , an Oboe 52 f , and an English Horn 52 g are shown in FIG. 5 a . Woodwind instruments are typically divided into 2 groups: flutes and reed instruments.

Flutes produce sound by directing a focused stream of air below the edge of a hole in a cylindrical tube. The flute family can be divided into two sub-families: open flutes and closed flutes. To produce a sound with an open flute, the player is required to blow a stream of air across a sharp edge that then splits the airstream. This split airstream then acts upon the air column contained within the flute's hollow causing it to vibrate and produce sound. To produce a sound with a closed flute, the player is required to blow air into a duct. This duct acts as a channel bringing the air to a sharp edge. As with the open flutes, the air is then split; this causes the column of air within the closed flute to vibrate and produce sound. Examples of this type of flute include the recorder, ocarina, and organ pipes.

Reed instruments produce sound by focusing air into a mouthpiece which then causes a reed, or reeds, to vibrate. Similarly to flutes, reed pipes are also further divided into two types: single reed and double reed. Single-reed woodwinds produce sound by fixing a reed onto the opening of a mouthpiece (using a ligature). When air is forced between the reed and the mouthpiece, the reed causes the air column in the instrument to vibrate and produce its unique sound. Single reed instruments include the clarinet, saxophone, and others such as the chalumeau. Double-reed instruments use two precisely cut, small pieces of cane bound together at the base. This form of sound production has been estimated to have originated in the middle to late Neolithic period; its discovery has been attributed to the observation of wind blowing through a split rush. The finished, bound reed is inserted into the instrument and vibrates as air is forced between the two pieces (again, causing the air within the instrument to vibrate as well). This family of reed pipes is subdivided further into another two sub-families: exposed double reed, and capped double reed instruments. Exposed double-reed instruments are played by having the double reed directly between the player's lips. This family includes instruments such as the oboe, cor anglais (also called English horn) and bassoon, and many types of shawms throughout the world. On the other hand, capped double-reed instruments have the double reed covered by a cap. The player blows through a hole in this cap that then directs the air through the reeds. This family includes the crumhorn.

Brass instrument. A brass instrument is a musical instrument that produces sound by sympathetic vibration of air in a tubular resonator in sympathy with the vibration of the player's lips. There are several factors involved in producing different pitches on a brass instrument. Slides, valves, crooks (though they are rarely used today), or keys are used to change vibratory length of tubing, thus changing the available harmonic series, while the player's embouchure, lip tension and air flow serve to select the specific harmonic produced from the available series.

Because the player of a brass instrument has direct control of the prime vibrator (the lips), brass instruments exploit the player's ability to select the harmonic at which the instrument's column of air vibrates. By making the instrument about twice as long as the equivalent woodwind instrument and starting with the second harmonic, players can get a good range of notes simply by varying the tension of their lips. Most brass instruments are fitted with a removable mouthpiece. Different shapes, sizes and styles of mouthpiece may be used to suit different embouchures, or to more easily produce certain tonal characteristics. Trumpets, trombones, and tubas are characteristically fitted with a cupped mouthpiece, while horns are fitted with a conical mouthpiece. One interesting difference between a woodwind instrument and a brass instrument is that woodwind instruments are non-directional. This means that the sound produced propagates in all directions with approximately equal volume. Brass instruments, on the other hand, are highly directional, with most of the sound produced traveling straight outward from the bell. This difference makes it significantly more difficult to record a brass instrument accurately. It also plays a major role in some performance situations, such as in marching bands. Pictorial views of a Trumpet 53 , a Tuba 53 a , a French Horn 53 b , and a Cornet 53 d are shown in FIG. 5 b.

Modern brass instruments generally come in one of two families: Valved brass instruments use a set of valves (typically three or four but as many as seven or more in some cases) operated by the player's fingers that introduce additional tubing, or crooks, into the instrument, changing its overall length. This family includes all of the modern brass instruments except the trombone: the trumpet, horn (also called French horn), euphonium, and tuba, as well as the cornet, flugelhorn, tenor horn (alto horn), baritone horn, sousaphone, and the mellophone. As valved instruments are predominant among the brasses today, a more thorough discussion of their workings can be found below. The valves are usually piston valves, but can be rotary valves; the latter are the norm for the horn (except in France) and are also common on the tuba.

Slide brass instruments use a slide to change the length of tubing. The main instruments in this category are the trombone family, though valve trombones are occasionally used, especially in jazz. The trombone family's ancestor, the sackbut, and the folk instrument bazooka are also in the slide family. There are two other families that have, in general, become functionally obsolete for practical purposes. Instruments of both types, however, are sometimes used for period-instrument performances of Baroque or Classical pieces. In more modern compositions, they are occasionally used for their intonation or tone color.

Percussion instrument. A percussion instrument is a musical instrument that is sounded by being struck or scraped by a beater including attached or enclosed beaters or rattles struck, scraped or rubbed by hand or struck against another similar instrument. The percussion section of an orchestra most commonly contains instruments such as the timpani, snare drum, bass drum, cymbals, triangle and tambourine, as well as keyboard percussion instruments such as the glockenspiel and xylophone. However, the section can also contain non-percussive instruments, such as whistles and sirens, or a blown conch shell. Percussive techniques can even be applied to the human body itself, as in body percussion. Percussion instruments are most commonly divided into two classes: Pitched percussion instruments, which produce notes with an identifiable pitch, and unpitched percussion instruments, which produce notes or sounds in an indefinite pitch. A pictorial view of a common drum set 54 is shown in FIG. 5 c , which includes a Floor tom 54 a , a Snare drum 54 b , Tom-toms 54 c , a Hi-hat 54 d , crash and ride cymbals 54 e , and a bass drum 54 f . Percussion instruments may play not only rhythm, but also melody and harmony. Because of the diversity of percussive instruments, it is not uncommon to find large musical ensembles composed entirely of percussion. Rhythm, melody, and harmony are all represented in these ensembles. Percussion instruments are classified by various criteria sometimes depending on their construction, ethnic origin, function within musical theory and orchestration, or their relative prevalence in common knowledge.

Piano. The piano is an acoustic, stringed musical instrument, in which the strings are struck by wooden hammers that are coated with a softer material (modern hammers are covered with dense wool felt; some early pianos used leather). It is played using a keyboard, which is a row of keys (small levers) that the performer presses down or strikes with the fingers and thumbs of both hands to cause the hammers to strike the strings. A piano usually has a protective wooden case surrounding the soundboard and metal strings, which are strung under great tension on a heavy metal frame. Pressing one or more keys on the piano's keyboard causes a wooden or plastic hammer (typically padded with firm felt) to strike the strings. The hammer rebounds from the strings, and the strings continue to vibrate at their resonant frequency. These vibrations are transmitted through a bridge to a soundboard that amplifies by more efficiently coupling the acoustic energy to the air. When the key is released, a damper stops the strings' vibration, ending the sound. Notes can be sustained, even when the keys are released by the fingers and thumbs, by the use of pedals at the base of the instrument. The sustain pedal enables pianists to play musical passages that would otherwise be impossible, such as sounding a 10-note chord in the lower register and then, while this chord is being continued with the sustain pedal, shifting both hands to the treble range to play a melody and arpeggios over the top of this sustained chord. Unlike the pipe organ and harpsichord, two major keyboard instruments widely used before the piano, the piano allows gradations of volume and tone according to how forcefully or softly a performer presses or strikes the keys.

Most modern pianos have a row of 88 black and white keys, 52 white keys for the notes of the C major scale (C, D, E, F, G, A and B) and 36 shorter black keys, which are raised above the white keys, and set further back on the keyboard. This means that the piano can play 88 different pitches (or “notes”), going from the deepest bass range to the highest treble. The black keys are for the “accidentals”, which are needed to play in all twelve keys. More rarely, some pianos have additional keys (which require additional strings). Most notes have three strings, except for the bass, which graduates from one to two. The strings are sounded when keys are pressed or struck, and silenced by dampers when the hands are lifted from the keyboard. There are three primary types of pianos: Grand, Upright, and Electronic.

Grands are the largest piano type, and frequently the most majestic (as well as expensive). Grand pianos are characterized by horizontal soundboards, which sometimes stretch up to 4 ft. (front to back). The soundboard is encased in a supportable opening platform that lifts upwards on the right. Dampers lie on top of the strings, adjacent to the hammers (also horizontal). The internal construction is braced with form-holders, usually made of wood, as well as the small equipped metal reinforcements. The casing is essentially “bottomless”, allowing one to see the soundboard support base, also of reinforced wood, which technically acts as the base. Keys consist of wood coated in ivory, or sometimes pure ivory, depending on the piano's manufacturers and classification. The grand piano has the standard 88 keys. Most of these pianos have sheet music stands. A retractable cover slides over, or folds down on the keys.

The baby grand piano, as the name implies, is essentially a smaller version of the regular grand piano. It has an 88-key set up, just like the regular grand does, but generally has a smaller soundboard and thus is not as loud as the regular grand.

Uprights are the most common type of acoustic piano and are a popular addition to a living room or parlor. The upright piano has been a favorite because it costs less, is more compact, and offers a warm sound. The soundboard is vertical, with strings that stretch downward and horizontal hammers and dampers. The hammers strike horizontally and are returned to resting position by springs, taking slightly longer than a grand piano's hammers (which are vertical and returned by the force of gravity). The support base of the soundboard is visible on the backside, as well as wooden reinforcements. Uprights usually cost less, depending on the model; however, some can exceed grands in total value. Although uprights often get depicted as inferior to the grand pianos, a five-foot upright can rival a typical grand in terms of tone quality and loudness. Like the grand, upright pianos vary in material construction.

Suitable for beginners or moving performers, electric pianos are usually the most affordable, and although they do not have the qualities of an acoustic, sound continues to improve for the high-end and mid-range instruments. They vary greatly in quality, some have hollow keys, while others try to replicate the feel and weight of acoustic keyboards. In addition to the features of an acoustic piano, electric pianos have a variety of sounds and settings such as organ, guitar, string, choir, and percussion. The numerous sounds on some keyboards make it virtually a portable band. Other pianos have limited functions, but this is better for someone who is trying to replicate an acoustic and save money. True electric pianos (compared to the plain keyboards) have a professional appearance and good materials (most consist largely of plastic), as well as touch-sensitive features and sometimes equipped frames. Most have connectors for pedals and computer interactive abilities. They never need to be tuned, and rapidly becoming more popular in modern bands. The electric piano also has the advantage of allowing the user to practice silently with headphones at times when doing so would otherwise disturb people. The few drawbacks include technological infancy and the requirement of a power supply.

Tempo. A ‘tempo’ is the speed or pace of a given piece, and is typically indicated with an instruction at the start of a piece (often using conventional Italian terms) and is usually measured in beats per minute (or bpm). In modern classical compositions, a “metronome mark” in beats per minute may supplement or replace the normal tempo marking, while in modern genres like electronic dance music, tempo will typically simply be stated in bpm. While the ability to hold a steady tempo is a vital skill for a musical performer, tempo is changeable. Depending on the genre of a piece of music and the performers' interpretation, a piece may be played with slight tempo rubato or drastic variances. In ensembles, the tempo is often indicated by a conductor or by one of the instrumentalists, for instance the drummer. While tempo is described or indicated in many different ways, including with a range of words (e.g., “Slowly”, “Adagio” and so on), it is typically measured in beats per minute (bpm or BPM). For example, a tempo of 60 beats per minute signifies one beat per second, while a tempo of 120 beats per minute is twice as rapid, signifying one beat every 0.5 seconds. The note value of a beat will typically be that indicated by the denominator of the time signature.

Tempo is not necessarily fixed throughout a musical piece. Within a piece (or within a movement of a longer work), a composer may indicate a complete change of tempo, often by using a double bar and introducing a new tempo indication, often with a new time signature and/or key signature.

Examples of common tempo markings, ranging from slowest to fastest tempo are: Larghissimo—very, very slowly (24 bpm and under); Adagissimo—very slowly; Grave—very slow (25-45 bpm), Largo—broadly (40-60 bpm); Lento—slowly (45-60 bpm); Larghetto—rather broadly (60-66 bpm); Adagio—slowly with great expression (66-76 bpm); Adagietto—slower than andante (72-76 bpm) or slightly faster than adagio (70-80 bpm); Andante—at a walking pace (76-108 bpm); Andantino—slightly faster than andante (although, in some cases, it can be taken to mean slightly slower than andante) (80-108 bpm); Marcia moderato—moderately, in the manner of a march (83-85 bpm); Andante moderato—between andante and moderato (thus the name) (92-98 bpm); Moderato—at a moderate speed (98-112 bpm); Allegretto—moderately fast (102-110 bpm); Allegro moderato—close to, but not quite allegro (116-120 bpm); Allegro—fast, quick, and bright (120-156 bpm) (molto allegro is slightly faster than allegro, but always in its range of 124-156 bpm); Vivace—lively and fast (156-176 bpm); Vivacissimo—very fast and lively (172-176 bpm); Allegrissimo or Allegro vivace—very fast (172-176 bpm); Presto—very, very fast (168-200 bpm); and Prestissimo—even faster than presto (200 bpm and over).

MIDI. Musical Instrument Digital Interface (MIDI) is a technical standard that describes a communications protocol, digital interface, and electrical connectors that connect a wide variety of electronic musical instruments, computers, and related audio devices for playing, editing and recording music. MIDI technology is standardized by the MIDI Manufacturers Association (MMA) and the MIDI Committee of the Association of Musical Electronics Industry (AMEI) in Tokyo. Version 1.0 of the MIDI standard is described in the standard entitled: “ The Complete MIDI 1.0 Detailed Specification Incorporating all Recommended Practices document ”, version 96.1 third edition, Published 2014 by The MIDI Manufacturers Association Los Angeles, Calif., U.S.A., and Version 2.0 of the standard is entitled: “ MIDI 2.0 Specification Overview, MIDI 2.0 Specification Overview MIDI Capability Inquiry ( MIDI - CI ) Common Rules for MIDI - CI Profiles, Common Rules for MIDI - CI Property Exchange, Universal MIDI Packet ( UMP ) Format, and MIDI 2.0 Protocol ” Version 1.0 Published Feb. 20, 2020 by Association of Musical Electronics Industry AMEI and MIDI Manufacturers Association MMA, and an overview of MIDI is described in a document entitled: “ What is MIDI ?” by Paul D. Lehrman, published 2017 by Tufts University, which are all incorporated in their entirety for all purposes as if fully set forth herein.

A single MIDI link through a MIDI cable can carry up to sixteen channels of information, each of which can be routed to a separate device or instrument. This could be sixteen different digital instruments, for example. MIDI carries event messages; data that specify the instructions for music, including a note's notation, pitch, velocity (which is heard typically as loudness or softness of volume); vibrato; panning to the right or left of stereo; and clock signals (which set tempo). When a musician plays a MIDI instrument, all of the key presses, button presses, knob turns and slider changes are converted into MIDI data. One common MIDI application is to play a MIDI keyboard or other controller and use it to trigger a digital sound module (which contains synthesized musical sounds) to generate sounds, which the audience hears produced by a keyboard amplifier. MIDI data can be transferred via MIDI or USB cable, or recorded to a sequencer or digital audio workstation to be edited or played back.

MIDI file. The Standard MIDI File (SMF) is a file format that provides a standardized way for music sequences to be saved, transported, and opened in other systems. The standard was developed and is maintained by the MMA, and usually uses a .mid extension. The compact size of these files led to their widespread use in computers, mobile phone ringtones, webpage authoring and musical greeting cards. These files are intended for universal use and include such information as note values, timing and track names. Lyrics may be included as metadata, and can be displayed by karaoke machines. SMFs are created as an export format of software sequencers or hardware workstations. They organize MIDI messages into one or more parallel tracks and timestamp the events so that they can be played back in sequence. A header contains the arrangement's track count, tempo and an indicator of which of three SMF formats the file uses. A type 0 file contains the entire performance, merged onto a single track, while type 1 files may contain any number of tracks that are performed in synchrony. Type 2 files are rarely used and store multiple arrangements, with each arrangement having its own track and intended to be played in sequence. Microsoft Windows bundles SMFs together with Downloadable Sounds (DLS) in a Resource Interchange File Format (RIFF) wrapper, as RMID files with a .rmi extension. RIFF-RMID has been deprecated in favor of Extensible Music Files (XMF).

MIDI Controller. A MIDI controller is any hardware or software that generates and transmits Musical Instrument Digital Interface (MIDI) data to MIDI-enabled devices, typically to trigger sounds and control parameters of an electronic music performance. MIDI controllers typically have some type of interface which the performer presses, strikes, blows or touches. This action generates MIDI data (e.g., notes played and their intensity), which can then be transmitted to a MIDI-compatible sound module or synthesizer using a MIDI cable. The sound module or synthesizer in turn produces a sound which is amplified through a loudspeaker. The most commonly used MIDI controller is the electronic musical keyboard MIDI controller. When the keys are played, the MIDI controller sends MIDI data about the pitch of the note, how hard the note was played and its duration. Other common MIDI controllers are wind controllers, which a musician blows into and presses keys to transmit MIDI data, and electronic drums. The MIDI controller can be populated with any number of sliders, knobs, buttons, pedals and other sensors, and may or may not include a piano keyboard.

MIDI Keyboard. A MIDI keyboard or controller keyboard is typically a piano-style electronic musical keyboard, often with other buttons, wheels and sliders, used for sending MIDI signals or commands over a USB or MIDI 5-pin cable to other musical devices or computers. MIDI keyboards lacking an onboard sound module cannot produce sounds themselves, however some models of MIDI keyboards contain both a MIDI controller and sound module, allowing them to operate independently. When used as a MIDI controller, MIDI information on keys or buttons the performer has pressed is sent to a receiving device capable of creating sound through modeling synthesis, sample playback, or an analog hardware instrument.

Musical piece. A musical piece, also referred to as musical composition, music composition, or simply composition, typically refers to an original piece or work of music, either vocal or instrumental, the structure of a musical piece, or to the process of creating or writing a new piece of music. People who create new compositions are called composers. Composers of primarily songs are usually called songwriters; with songs, the person who writes lyrics for a song is the lyricist. In general, composition consists in two things—The first is the ordering and disposing of several sounds in such a manner that their succession pleases the ear, and the second is the rendering audible of two or more simultaneous sounds in such a manner that their combination is pleasant.

The most popular music genres are classical music and popular music. While in popular music, chords are much more predictable and repetitive than in classical music. For example, a typical Bach piece uses dozens of different chords in amazingly unique combinations. Further, popular tunes generally use a lot of repeated notes, short repeated melodic phrases, and very simple melodic lines. Songs are easy to learn, easy to remember, and simple and fun to sing or play. Classical melodies have a much more complex structure, tend to have longer repeated phrases, and can be much more challenging, and more rewarding, to learn and perform. The main advantage popular music has over classical music is that pop music tends to be more rhythmically sophisticated. Drum patterns in pop music layer different rhythms for bass drum, snare drum, and hi hat. Most classical music is typically not as rhythmically interesting. For instance, some Bach pieces consist of nothing but straight sixteenth notes.

Popular music. Popular music is music with wide appeal that is typically distributed to large audiences through the music industry. These forms and styles can be enjoyed and performed by people with little or no musical training. Typically, popular music, unlike art music, is conceived for mass distribution to large and often socioculturally heterogeneous groups of listeners, is stored and distributed in non-written form, is only possible in an industrial monetary economy where it becomes a commodity and is commonly subject to commercial considerations. Popular music is found on most commercial and public service radio stations, in most commercial music retailers and department stores, and movie and television soundtracks.

Common genres of popular music include ‘Rock music’ that originated as “rock and roll” in the United States in the early 1950s, and developed into a range of different styles in the 1960 s and later, particularly in the United Kingdom and in the United States, Electronic music employs electronic musical instruments, digital instruments and circuitry-based music technology, ‘Soul music’ that originated in the African American community throughout the United States in the 1950 s and early 1960s and combines elements of African-American gospel music, rhythm and blues and jazz, ‘Funk music’ that de-emphasizes melody and chord progressions and focuses on a strong rhythmic groove of a bassline played by an electric bassist and a drum part played by a drummer, often at slower tempos than other popular music, ‘Country music’ that often consists of ballads and dance tunes with generally simple forms, folk lyrics, and harmonies mostly accompanied by string instruments such as banjos, electric and acoustic guitars, steel guitars (such as pedal steels and dobros), and fiddles as well as harmonicas, ‘Latin music’ that comes from Spanish- and Portuguese-speaking areas of the world, ‘Reggae music’ that incorporates different stylistic techniques form rhythm and blues, jazz, African, Caribbean, and other genres as well but what makes reggae unique are the vocals and lyrics, ‘Hip-hop music’ that is broadly defined as a stylized rhythmic music that commonly accompanies rapping, a rhythmic and rhyming speech that is chanted, and ‘Polka music’ that is a dance music.

Classical music. Examples of classical music forms are Aria—which refers to a long accompanied song for a solo voice, typically one in an opera or oratorio, and is typically a formal musical composition that is a self-contained piece for one voice, with or without instrumental or orchestral accompaniment, normally part of a larger work; Cadenza—which refers to a virtuoso solo passage inserted into a movement in a concerto or other work, typically near the end, and generally includes an improvised or written-out ornamental passage played or sung by a soloist or soloists, usually in a “free” rhythmic style, and often allowing virtuosic display, while during this time the accompaniment will rest, or sustain a note or chord; Concerto—which is a musical composition for a solo instrument or instruments accompanied by an orchestra, especially one conceived on a relatively large scale, and includes an instrumental composition, written for one or more soloists accompanied by an orchestra or other ensemble. The typical three-movement structure, a slow movement (e.g., lento or adagio) preceded and followed by fast movements (e.g. presto or allegro); Chamber music—which is an instrumental music played by a small ensemble, with one player to a part, the most important form being the string quartet which developed in the 18th century, and typically is a form of classical music that is composed for a small group of instruments—traditionally a group that could fit in a palace chamber or a large room. Most broadly, it includes any art music that is performed by a small number of performers, with one performer to a part (in contrast to orchestral music, in which each string part is played by a number of performers); Movement—which is a principal division of a longer musical work, self-sufficient in terms of key, tempo, and structure, and while individual or selected movements from a composition are sometimes performed separately, a performance of the complete work requires all the movements to be performed in succession; Sonata—which is a composition for an instrumental soloist, often with a piano accompaniment, typically in several movements with one or more in sonata form; Opera—which is a dramatic work in one or more acts, set to music for singers and instrumentalists, and where the music has a leading role and the parts are taken by singers, but is distinct from musical theatre, typically involving a collaboration between a composer and a librettist and incorporating a number of the performing arts, such as acting, scenery, costume, and sometimes dance or ballet; and Overture—which is an orchestral piece at the beginning of an opera, suite, play, oratorio, or other extended composition, and Symphony—which is an elaborate musical composition for full orchestra, typically in four movements, at least one of which is traditionally in sonata form.

Vocal music. Vocal music is a type of singing performed by one or more singers, either with instrumental accompaniment, or without instrumental accompaniment (a cappella), in which singing provides the main focus of the piece. Music which employs singing but does not feature it prominently is generally considered to be instrumental music as is music without singing. Music without any non-vocal instrumental accompaniment is referred to as a cappella. Vocal music typically features sung words called lyrics, although there are notable examples of vocal music that are performed using non-linguistic syllables, sounds, or noises, sometimes as musical onomatopoeia, such as jazz scat singing. A short piece of vocal music with lyrics is broadly termed a song, although in different styles of music, it may be called an aria or hymn.

Vocal music often has a sequence of sustained pitches that rise and fall, creating a melody, but some vocal styles use less distinct pitches, such as chants or a rhythmic speech-like delivery, such as rapping. There are extended vocal techniques that may be used, such as screaming, growling, throat singing, or yodelling.

Chroma features. Chroma features or chromagram, also referred to as “pitch class profiles”, relates to the twelve different pitch classes, and are a powerful tool for analyzing music whose pitches can be meaningfully categorized (often into twelve categories) and whose tuning approximates to the equal-tempered scale. One main property of chroma features is that they capture harmonic and melodic characteristics of music, while being robust to changes in timbre and instrumentation. The underlying observation is that humans perceive two musical pitches as similar in color if they differ by an octave. Based on this observation, a pitch can be separated into two components, which are referred to as tone height and chroma. Assuming the equal-tempered scale, one considers twelve chroma values represented by the set {C, C♯, D, D♯, E, F, F♯, G, G♯, A, A♯, B} that consists of the twelve pitch spelling attributes as used in Western music notation. Note that in the equal-tempered scale different pitch spellings such C♯ and db refer to the same chroma. Enumerating the chroma values, one can identify the set of chroma values with the set of integers {1, 2, . . . , 12}, where 1 refers to chroma C, 2 to C♯, and so on. A pitch class is defined as the set of all pitches that share the same chroma. For example, using the scientific pitch notation, the pitch class corresponding to the chroma C is the set { . . . , C−2, C−1, C0, C1, C2, C3 . . . } consisting of all pitches separated by an integer number of octaves. Given a music representation (e.g. a musical score or an audio recording), the main idea of chroma features is to aggregate for a given local time window (e.g. specified in beats or in seconds) all information that relates to a given chroma into a single coefficient. Shifting the time window across the music representation results in a sequence of chroma features each expressing how the representation's pitch content within the time window is spread over the twelve chroma bands. The resulting time-chroma representation is also referred to as chromagram. The figure above shows chromagrams for a C-major scale, once obtained from a musical score and once from an audio recording. Because of the close relation between the terms chroma and pitch class, chroma features are also referred to as pitch class profiles.

A chroma toolbox is presented in a paper entitled: “ CHROMA TOOLBOX: MATLAB IMPLEMENTATIONS FOR EXTRACTING VARIANTS OF CHROMA - BASED AUDIO FEATURES ” by Meinard Muller and Sebastian Ewert, published 2011 by the International Society for Music Information Retrieval, which is incorporated in its entirety for all purposes as if fully set forth herein. Chroma-based audio features, which closely correlate to the aspect of harmony, are a well-established tool in processing and analyzing music data. There are many ways of computing and enhancing chroma features, which results in a large number of chroma variants with different properties. The toolbox contains MATLAB implementations for extracting various types of recently proposed pitch-based and chroma-based audio features. Providing the MATLAB implementations on a well-documented website under a GNU-GPL license, our aim is to foster research in music information retrieval. As another goal, we want to raise awareness that there is no single chroma variant that works best in all applications. To this end, we discuss two example applications showing that the final music analysis result may crucially depend on the initial feature design step.

Music Arrangement. As used herein, the term “arrangement” in the context of ‘music arrangement’ refers to any reworking of a musical piece (or part thereof), that may be any composition or any original piece of music. In one example, an arrangement may refer to adapted version of a musical piece, such as adaptation of the corresponding sheet music, so that it can be played by a different instrument or combination of instruments from the original. For example, a song written for one voice with piano accompaniment might be arranged so that it can be sung in parts by a choir, or a piece for violin might be arranged so that it can be played on a clarinet instead. In another example, an arrangement may refer to adapted version of a musical piece, such as simplifying or elaboration of the corresponding sheet music, so that it can be played by less skilled players or beginners, while retaining the general character of the original musical piece.

A statistical-modeling method for piano reduction, such as converting an ensemble score into piano scores that can control performance difficulty, is presented in an article entitled: “ Statistical piano reduction controlling performance difficulty ” by Eita Nakamura and Kazuyoshi Yoshii [doi:10.1017/ATSIP.2018.18], published 2018 in SIP (2018), vol. 7, e13, which is incorporated in its entirety for all purposes as if fully set forth herein. While previous studies have focused on describing the condition for playable piano scores, it depends on player's skill and can change continuously with the tempo. The article describes computationally quantifying performance difficulty as well as musical fidelity to the original score, and formulate the problem as optimization of musical fidelity under constraints on difficulty values. First, performance difficulty measures are developed by means of probabilistic generative models for piano scores and the relation to the rate of performance errors is studied. Second, to describe musical fidelity, we construct a probabilistic model integrating a prior piano-score model and a model representing how ensemble scores are likely to be edited. An iterative optimization algorithm for piano reduction is developed based on statistical inference of the model. We confirm the effect of the iterative procedure; we find that subjective difficulty and musical fidelity monotonically increase with controlled difficulty values; and we show that incorporating sequential dependence of pitches and fingering motion in the piano-score model improves the quality of reduction scores in high-difficulty cases.

An automatic arrangement system for piano reduction that arranges music algorithmically for the piano while considering various roles of the piano in music is presented in an article entitled: “ Automatic System for the Arrangement of Piano Reductions ” by Shih-Chuan Chiu, Man-Kwan Shan, and Jiun-Long Huang, published 2009 in the 11th IEEE International Symposium on Multimedia, which is incorporated in its entirety for all purposes as if fully set forth herein. The automatic arrangement system for piano reduction is achieved by first analyzing the original music in order to determine the type of arrangement element performed by an instrument. Then each phrase is identified and is associated with a weighted importance value. At last, a phrase selection algorithm is proposed to select phrases with maximum importance to arrangement under the constraint of piano playa bility. Our experiments demonstrate that the proposed system has the ability to create piano arrangement.

Looking for a piano sheet music with proper difficulty for a piano learner is always an important work to his/her teacher. A study on a new and challenging issue of recognizing the difficulty level of piano sheet music is described in an article entitled: “ A Study on Difficulty Level Recognition of Piano Sheet Music ” by Shih-Chuan Chiu and Min-Syan Chen, published December 2012 in the ISM '12: Proceedings of the 2012 IEEE International Symposium on Multimedia, which is incorporated in its entirety for all purposes as if fully set forth herein. To analyze the semantic content of music, we focus on symbolic music, i.e., sheet music or score. Specifically, difficulty level recognition is formulated as a regression problem to predict the difficulty level of piano sheet music. Since the existing symbolic music features are not able to capture the characteristics of difficulty, we propose a set of new features. To improve the performance, a feature selection approach, RReliefF, is used to select relevant features. An extensive performance study is conducted over two real datasets with different characteristics to evaluate the accuracy of the regression approach for predicting difficulty level. The best performance evaluated in terms of the R2 statistics over two datasets reaches 39.9% and 38.8%, respectively.

Huge sheet music collections exist on the Web, allowing people to access public domain scores for free. However, beginners may be lost in finding a score appropriate to their instrument level, and should often rely on themselves to start out on the chosen piece. In this instrumental e-Learning context, a Score Analyzer prototype in order to automatically extract the difficulty level of a MusicXML piece and suggest advice thanks to a Musical Sign Base (MSB) is proposed in an article entitled: “ SCORE ANALYZER: AUTOMATICALLY DETERMINING SCORES DIFFICULTY LEVEL FOR INSTRUMENTAL E - LEARNING ” by Véronique Sebastien, Henri Ralambond rainy, Olivier Sébastien, and Noel Conruyt of IREMIA—Laboratoire d'Informatique et de Mathématiques, EA2525 University of Reunion Island, Saint-Denis, Reunion (FRANCE), published October 2012 in 13th International Society for Music Information Retrieval Conference (ISMIR 2012), which is incorporated in its entirety for all purposes as if fully set forth herein. The article first review methods related to score performance information retrieval, and then identifies seven criteria to characterize technical instrumental difficulties and propose methods to extract them from a MusicXML score. The relevance of these criteria is then evaluated through a Principal Components Analysis and compared to human estimations. Lastly, the article discusses the integration of this work to @-MUSE, a collaborative score annotation platform based on multimedia contents indexation.

In the Western classical tradition, musicians play music from notated sheet music, called a score. When playing music from a score, a musician translates its visual symbols into sequences of instrument-specific physical motions. Hence, a music score's overall complexity represents a sum of the cognitive and mechanical acuity required for its performance. For a given instrument, different notes, intervals, articulations, dynamics, key signatures, and tempo represent dissimilar levels of difficulty, which vary depending on the performer's proficiency. Individual musicians embrace this tenet, but may disagree about the degrees of difficulty. A ‘musiplectics’, a systematic and objective approach to computational assessment of the complexity of a music score for any instrument, is introduced in a paper entitled: “ Musiplectics: Computational Assessment of the Complexity of Music Scores ” by Ethan Holder, Eli Tilevich, and Amy Gillick, published October 2015 in ONWARD '15 [ACM 978-1-4503-1995-9/13/10, http://dx.doi.org/10.1145/2508075.2514879], which is incorporated in its entirety for all purposes as if fully set forth herein. Musiplectics defines computing paradigms for automatically and accurately calculating the complexity of playing a music score on a given instrument. The core concept codifies a two-phase process. First, music experts rank the relative difficulty of individual musical components (e.g., notes, intervals, dynamics, etc.) for different playing proficiencies and instruments. Second, a computing engine automatically applies this ranking to music scores and calculates their respective complexity. As a proof of concept of musiplectics, we present an automated, Web-based application called Musical Complexity Scoring (MCS) for music educators and performers. Musiplectics can engender the creation of practical computing tools for objective and expeditious assessment of a music score's suitability for the abilities of intended performers.

While the difficulty of the music can be classified by a variety of standard, conventional methods are classified by the subjective judgment based on the experience of many musicians or conductors. Music score is difficult to evaluate as there is no quantitative criterion to determine the degree of difficulty. A new classification method for determining the degree of difficulty of the music is proposed in a paper entitled: “ A Method for Measuring the Difficulty of Music Scores ” by Yang-Eui Song and Yong Kyu Lee [www.ksci.re.kr http://dx.doi.org/10.9708/jksci.2016.21.4.039] published April 2016 in the Journal of The Korea Society of Computer and Information Vol. 21 No. 4, which is incorporated in its entirety for all purposes as if fully set forth herein. In order to determine the degree of difficulty, we convert the score, which is expressed as a traditional music score, into electronic music sheet. Moreover, we calculate information about the elements needed to play sheet music by distance of notes, tempo, and quantifying the ease of interpretation. Calculating a degree of difficulty of the entire music via the numerical data, we suggest the difficulty evaluation of the score, and show the difficulty of music through experiments.

Commercial application for writing music or songs are available, such as Ludwig (http://www.write-music.com/), which is a music software for Windows that helps you to write your own songs, while you simply enter or play a melody. Ludwig does the rest: he finds the proper chords and writes all parts of a professionally sounding band. It was never easier to compose, arrange or accompany songs, and AnthemScore (https://www.lunaverus.com/), which is the leading software for automatic music transcription. Convert mp3, way, and other audio formats into sheet music using a neural network trained on millions of data samples. Use powerful editing tools to tweak notes, beats, and time signatures. Print or save as PDF, MIDI, or XML.

Online music learning. Increasing Internet penetration, smartphones and tablets in the modern and emerging economies are boosting the growth of the online music learning market, as an alternative to the traditional personal one-on-one piano lessons. Various online courses and individual lessons that offer enhanced music skills are provided as applications for mobile phones and tablets, typically based on Windows, iOS, Android, and MacOS platforms, are offered by multiple key players, supporting various musical instruments such as guitar, ukulele, piano, vocal, drums, and bass. In addition to the reduced costs, such online music learning provides an interactive and fun experience, while providing more flexibility and being less bound by location and time, compared to personal based tutoring. A pictorial illustration of an online music learning is described in a view 60 c in FIG. 6 c , showing a user 36 that plays a piano 83 according to musical symbols visualized poon a display as part of a tablet 35 a.

Methods and apparatus that help users learn to play one or more instruments by employing simplified note recognition techniques, improved feedback, and a simplified keyboard tablature are provided in U.S. Pat. No. 7,030,307 to Wedel entitled: “Music teaching device and method”, which is incorporated in its entirety for all purposes as if fully set forth herein. In one aspect, users are encouraged to play specified notes within a specified time frame. The pitch and duration can be represented in any suitable manner, but are advantageously displayed together as a single icon ( 232 A- 232 F). Preferably, the icon has an elongated shape in which the horizontal length correlates with duration of the note, and its vertical position on a display correlates with a pitch of the note. Notes can thus be represented by elongated lines, bars, ellipses, or even missiles or arrows. In another aspect, an improved musical tablature effectively clusters juxtaposed black keys (412A, 412C, 412E) and white keys (412B, 412D) on a display for easier visualization.

A musical keyboard that is connected to a computer is described in U.S. Pat. No. 7,174,510 to Salter entitled: “Interactive game providing instruction in musical notation and in learning an instrument”, which is incorporated in its entirety for all purposes as if fully set forth herein. The computer implements a graphical user interface for teaching users to play the musical instrument. A computer readable music file, such as a MIDI file, is used to drive the creation of game objects that travel from a point of origination along a path toward a key of a virtual keyboard. In one form, when a user presses a key of the musical keyboard within a certain time window of arrival of the game object at the corresponding key of the virtual keyboard, the user is awarded with an audio presentation, a visual presentation and/or with game points. In a more structured learning mode, the game can be played with selectable, progressively more difficult challenges that the user masters on the road to proficiency.

A system and method for improving musical education through use of a game is disclosed in U.S. Pat. No. 9,492,756 to Izkovsky et al. entitled: “System and method for analyzing a digitalized musical performance”, which is incorporated in its entirety for all purposes as if fully set forth herein. The method includes the steps of: receiving electrical signals associated with a musical piece provided by a user of the game; converting the electrical signals into digital samples; and analyzing the digital samples with use of auxiliary information, for purposes of improving signal analysis accuracy, resulting in determining various parameters of the musical piece provided by the user, wherein the auxiliary information is a-priori data related to at least one element selected from the group consisting of a musical instrument being played, a technical environment, the game, and information regarding the user of the game. Visual and/or audio feedback may also be provided to the user regarding the musical piece that they are providing.

A system and method for providing exercise in playing a music instrument is disclosed in U.S. Pat. No. 9,218,748 to Kaipainen et al. entitled: “System and method for providing exercise in playing a music instrument”, which is incorporated in its entirety for all purposes as if fully set forth herein. With prior art solutions there is a common problem of inadequate motivation of the user to continue practicing, and inadequate learning of items which are difficult for a specific user. The present solution detects characteristics of the user's play detected and uses them to provide a suitable program of exercises and to provide feedback which enhances motivation of the user.

A system for providing a user a virtual exercise in playing a music instrument relative to the user's skill characteristics is disclosed in U.S. Pat. No. 9,767,705 to Klapuri et al. entitled: “System for estimating user's skill in playing a music instrument and determining virtual exercises thereof”, which is incorporated in its entirety for all purposes as if fully set forth herein. The system includes: a processing entity and a memory entity for processing and storing data, respectively to execute the system functions, and a data transfer entity for receiving and transmitting data, the system configured to: obtain musical notation data, analyze it to assign the musical piece to which such data pertains a number of difficulty characteristics with scalar values, provide the user with a number of musical pieces, with known difficulty characteristics, as virtual exercises to be completed by playing an instrument, obtain user performance data of completed virtual exercises, analyze the user performance data to determine and assign the user with a number of skill characteristics values in accordance with the difficulty characteristic values of the completed musical pieces, and determine a musical piece for the user as a virtual exercise.

A system and method for providing exercise in playing a music instrument is disclosed in U.S. Pat. No. 9,218,748 to Kaipainen et al. entitled: “System and method for providing exercise in playing a music instrument”, which is incorporated in its entirety for all purposes as if fully set forth herein. With prior art solutions there is a common problem of inadequate motivation of the user to continue practicing, and inadequate learning of items which are difficult for a specific user. The present solution detects characteristics of the user's play detected and uses them to provide a suitable program of exercises and to provide feedback which enhances motivation of the user.

Providing real-time interaction between a first player and a second player to collaborate for a musical performance over a network is disclosed in U.S. Pat. No. 10,182,093 to Klapuri entitled: “Computer implemented method for providing real-time interaction between first player and second player to collaborate for musical performance over network”, which is incorporated in its entirety for all purposes as if fully set forth herein. The providing includes maintaining a reference data item generated using a musical instrument performed by a user; receiving first user input data generated using a first musical instrument, the received first user input data associated with the first musical instrument and the musical performance; receiving second user input data generated using a second musical instrument, the received second user input data associated with the second musical instrument and the musical performance; detecting a missing data packet within the received second user input data when generating real-time collaboration data for the musical performance; replacing the missing data packet using the reference data item to correct the second user input data; and generating real-time collaboration data for the musical performance based on the corrected second user input data.

A system and method for providing exercise in playing a music instrument is disclosed in U.S. Pat. No. 9,218,748 to Kaipainen et al. entitled: “System and method for providing exercise in playing a music instrument”, which is incorporated in its entirety for all purposes as if fully set forth herein. With prior art solutions there is a common problem of inadequate motivation of the user to continue practicing, and inadequate learning of items which are difficult for a specific user. The present solution detects characteristics of the user's play detected and uses them to provide a suitable program of exercises and to provide feedback which enhances motivation of the user.

An apparatus, system and method are disclosed for teaching musical instruction is described in U.S. Pat. No. 9,721,479 to Citron et al. entitled: “Apparatus, system and method for teaching music and other art forms”, which is incorporated in its entirety for all purposes as if fully set forth herein. The invention disclosed includes providing music instruction based upon a student's ability and preferred most dynamic efficient method of learning.

A computer implemented method for providing feedback of harmonic content relating to a music track is disclosed in U.S. Pat. No. 10,235,898 to Ryynanen et al. entitled: “Computer implemented method for providing feedback of harmonic content relating to music track”, which is incorporated in its entirety for all purposes as if fully set forth herein. The method comprising: receiving music track information; generating harmonic music track parameters based on the received music track information; displaying notation information for a user for performing the music track at a given time for the music track based on the harmonic music track parameters; receiving harmonic user content generated by an instrument performed by the user, using at least one capturing device; generating real-time performance feedback for the user based on comparison of the harmonic user content and the harmonic music track parameters according to predefined settings; receiving reference harmonic user content from a plurality of reference users over a public network; adjusting, based on the reference harmonic user content, at least one of the following: the predefined settings; and the harmonic music track parameters.

Real-time jamming that is automatically assisted for musicians is disclosed in U.S. Pat. No. 10,504,498 to Ryynänen et al. entitled: “Real-time jamming assistance for groups of musicians”, which is incorporated in its entirety for all purposes as if fully set forth herein. A real-time audio signal is received of played music that is played by at least one person. Beat is tracked of the played music from the real-time audio signal and accordingly a time of a next beat is predicted. At least one of chords; notes; and drum sounds is recognized from the real-time audio signal and repetitions in the played music are accordingly detected. A next development is predicted in the played music, based on the detected repetitions, including at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat. A real-time output is produced based on the predicted next development in the played music.

The Learner Interaction Monitoring Systems (LiMS) is a web-based application that can interface with any web-based course delivery platform to transform the online learning environment into an active observer of learner engagement, and is disclosed in U.S. Pat. No. 10,490,096 to Sorenson et al. entitled: “Learner interaction monitoring system”, which is incorporated in its entirety for all purposes as if fully set forth herein. The LiMS ‘event capture model’ collects detailed real-time data on learner behavior in self-directed online learning environments, and interprets these data by drawing on behavioral research. The LiMS offers education and training managers in corporate contexts a valuable tool for the evaluation of learner performance and course design. By allowing more detailed demonstration of ROI in education and training, LiMS allows managers to make the case for web based courseware that reflects appropriate and evidence-based instructional design, rather than budgetary constraints.

A device (1) for monitoring the use accuracy of percussion instruments (5) is disclosed in U.S. Patent Application No. 2015/0310841 to SEMENZATO entitled: “Device for monitoring use accuracy of percussion instruments”, which is incorporated in its entirety for all purposes as if fully set forth herein. The device includes a sensor (3) intended to detect musical data (T, I) from said instrument (5), an electronic device (9) adapted to sample at a sampling frequency Fc the musical data (T, I) detected by said sensor (3), which comprise the instant (T), detected with respect to an initial instant TO taken as a reference, and the intensity (I) of each stroke that reaches the percussion instrument (5) during an execution of a musical score (PMS) played by a user (100), and adapted to store said musical data (T, I) in a memory device (11). An external computing device (19) having a database (50) containing at least one musical sample score (PMC) intended to be considered as a reference score, or a microprocessor (6) comprised in the electronic device (9), compares the musical data (T, I) detected by the sensor (3) during the execution of a played musical score (PMS) by a user (100) with the corresponding musical data (Tc, Ic) of the musical sample score (PMC), and provides the deviation between the measured musical data (T, I) of the played musical score (PMS) and the corresponding ones (Tc, Ic) of the musical sample score (PMC).

Systems and methods capable of providing adaptive and responsive accompaniment to music with fixed chord progressions, such as jazz and pop, are disclosed in U.S. Pat. No. 10,032,443 to Braasch et al. entitled: “Interactive, expressive music accompaniment system”, which is incorporated in its entirety for all purposes as if fully set forth herein. A system can include one or more sound-capturing devices, a signal analyzer to analyze captured sound signals, and an electronic sound-producing component that produces electronic sounds as an accompaniment.

A system (100) for teaching a user to play a musical instrument (114) from musical notation via virtual exercises is described in Patent Cooperation Treaty (PCT) International Publication Number WO 2017/037342 by LEHTONEN et al. entitled: “System for teaching a user to play a musical instrument from musical notation via virtual exercises and a method thereof”, which is incorporated in its entirety for all purposes as if fully set forth herein. The system comprising: at least one electronic device (102) comprising at least a processing entity (104), a memory entity (106) and a display (108), means for forming or receiving at least one play signal (110) produced by a user on a musical instrument (114), the processing entity (104) being arranged to provide at least graphical musical notation content, such as a note or a chord, via the display (108), the processing entity (104) being further arranged to obtain at least play signal data via said means for forming or receiving at least one play signal (110), the processing entity (104) being further arranged to execute audio recognition to recognize the play signal data and compare it with data relating to the presented graphical content to at least determine if the play signal corresponds to said graphical content, the processing entity (104) being further arranged to assign a score to represent the result of said comparison and store said score, the processing entity (104) being further arranged to present the score to the user via the display (108), and to determine, at least on the basis of said score and other stored scores, further at least graphical musical notation content, to be presented to the user via the display (108). Corresponding method and computer program product are also presented.

Invention that relates to the field of audio recognition, in particular to computer implemented note recognition methods in a gaming application, is disclosed in U.S. Pat. No. 9,711,121 to Ahmaniemi entitled: “Latency enhanced note recognition method in gaming”, which is incorporated in its entirety for all purposes as if fully set forth herein. Furthermore, the invention relates to improving latency of such audio recognition methods. One of the embodiments of the invention described herein is a method for note recognition of an audio source. The method includes: dividing an audio input into a plurality of frames, each frame having a pre-determined length, conducting a frequency analysis of at least a set of the plurality of frames, based on the frequency analysis, determining if a frame is a transient frame with a frequency change between the beginning and end of the frame, comparing the frequency analysis of each said transient frame to the frequency analysis of an immediately preceding frame and, based on said comparison, determining at least one probable pitch present at the end of each transient frame, and for each transient frame, outputting pitch data indicative of the probable pitch present at the end of the transient frame.

A workstation system that produces a display presentation of a selected performance composition (e.g., a musical composition) responsive to composition data and responsive to one or both of input variables and a selected operating mode is disclosed in U.S. Pat. No. 7,157,638 to Sitrick entitled: “System and methodology for musical communication and display”, which is incorporated in its entirety for all purposes as if fully set forth herein. The workstation can communicate with one or more external devices, such as other workstations, etc. The display system provides for selection of original compositions, creation of derivative compositions, distribution of compositions, monitoring of each performer's performance, group virtual performances, and for local and distributed retrieval and editing, which for music includes things such as changing keys, pitch, tempo, and other parameters. The musical composition's transformation can be performed locally or at the central or distributed music database. The musical composition data can be transposed via a controller, and can be transmitted to a plurality of the individual workstations that then display the musical composition. In one embodiment, a display system for use by a plurality of users provides a plurality of display presentations of a selected musical composition. The system is comprised of a plurality of individual workstations, each workstation comprising a communication interface providing for communications with the respective workstation of music data representative of the selected musical composition, memory for locally storing the data responsive to the communications interface, and a display apparatus provides a local visual display presentation representative of the selected musical composition responsive to the stored data. The system further provides for synchronizing the presentation on the plurality of local visual display presentations of the selected musical composition.

An interactive game designed for learning to play a guitar is disclosed in U.S. Pat. No. 8,986,090 to EPSTEIN entitled: “Interactive guitar game designed for learning to play the guitar”, which is incorporated in its entirety for all purposes as if fully set forth herein. A guitar may be connected to a computer or other platform, capable of loading music and displaying notes and chords and other feedback and visual learning aids on a display screen, allowing a user to read music and play along. The goal of the software or interactive game engine is for players to learn how to play a guitar. Users may operate the game in a number of modes with different goals, playing mini-games throughout the levels of the game. The game provides feedback and statistics to help users learn how to play the guitar.

A system and method for learning, composing, accessing and playing music is disclosed in in Patent Cooperation Treaty (PCT) International Publication Number WO 2015/113360 by Shi entitled: “System and method for learning, composing, and playing music with physical objects”, which is incorporated in its entirety for all purposes as if fully set forth herein. The system includes a plurality of physical objects each includes an identifier and assigned a music-related indicator. The system further includes an interactive surface configured to recognize the identifier and location information relative to the interactive surface of a physical object placed on top of the interactive surface. Upon a plurality of objects being placed on the interactive surface to form a structural pattern, the processor is configured to derive a music piece from the structural pattern.

A system is disclosed in U.S. Pat. No. 7,893,337 to Lenz entitled: “System and method for learning music in a computer game”, which is incorporated in its entirety for all purposes as if fully set forth herein. The system comprising means for receiving a first input from an electronic device, said first input pertaining to performance of music by a user, means for receiving a second input, said second input pertaining at least to music intended to be performed by the user; and a comparison module software executing on a computer and adapted to receive said first input and to receive said second input, wherein the comparison module compares the first input from a user to the second input to produce at least one indicia of the user's success in performing the intended music correctly, the comparison module sends to a display module associated with the user information including at least the music intended to be performed by the user and the indicia of the user's success in performing the intended music correctly, and the comparison module further sends to the display module associated with the user a timing signal suitable for indicating the speed at which the music should be shown on the display and played by the user, said timing signal computed according to one or more tempo modes selectable by the user, is disclosed.

A music practice feedback method, system, and non-transitory computer readable medium including a displaying device configured to display sheet music, a collecting device configured to collect information related to a playing of the sheet music by a plurality of players, and a display changing device configured to change a display of the sheet music based on said collected information, are disclosed in U.S. Pat. No. 9,672,799 to Kim et al. entitled: “Music practice feedback system, method, and recording medium”, which is incorporated in its entirety for all purposes as if fully set forth herein.

A self-adjusting music scrolling method is disclosed in U.S. Pat. No. 7,482,529 to Flamini et al. entitled: “Self-adjusting music scrolling system”, which is incorporated in its entirety for all purposes as if fully set forth herein. The self-adjusting music method comprises providing a display screen, selecting a music score to be played by a musician, wherein the music score is cataloged and stored in a music score database as a first MIDI file, displaying a first portion of the selected music score on the display screen, recording musical notes played by a musician with a digital device, storing the recorded musical notes in memory as a WAV file, converting the WAV file into a second MIDI file, comparing the first MIDI file and the second MIDI file with a MIDI comparison algorithm, determining if the first MIDI file substantially matches the second MIDI file, automatically adjusting the music score on the display screen to show a second portion of the selected music score upon determining that the first MIDI file substantially matches the second MIDI file, and displaying one or more mistakes detected on the display screen upon determining that the first MIDI file does not substantially match the second MIDI file.

A system and method in a building or vehicle for an actuator operation in response to a sensor according to a control logic are disclosed in U.S. Patent Application Publication No. 2013/0201316 to Binder et al. entitled: “System and method for server based control”, which is incorporated in its entirety for all purposes as if fully set forth herein. The system comprising a router or a gateway communicating with a device associated with the sensor and a device associated with the actuator over in-building or in-vehicle networks, and an external Internet-connected control server associated with the control logic implementing a PID closed linear control loop and communicating with the router over external network for controlling the in-building or in-vehicle phenomenon. The sensor may be a microphone or a camera, and the system may include voice or image processing as part of the control logic. A redundancy is used by using multiple sensors or actuators, or by using multiple data paths over the building or vehicle internal or external communication. The networks may be wired or wireless, and may be BAN, PAN, LAN, WAN, or home networks.

A method to visually detect and recognize fingering gestures of the left hand of a guitarist is described in an article by Anne-Marie Burns and Marcelo Wanderley entitled: “ Visual Methods for the Retrieval of Guitarist Fingering ”, published Jun. 4-8, 2006 in a Conference Paper of “ Proceedings of the 2006 International Conference on New Interfaces for Musical Expression ( NIME 06)”, Paris, France, which is incorporated in its entirety for all purposes as if fully set forth herein. The method has been developed following preliminary manual and automated analysis of video recordings. These first analyses led to some important findings about the design methodology of a vision system for guitarist fingering, namely the focus on the effective gesture, the consideration of the action of each individual finger, and a recognition system not relying on comparison against a knowledge base of previously learned fingering positions. Motivated by these results, studies on three aspects of a complete fingering system were conducted: the first on finger tracking; the second on strings and frets detection; and the last one on movement segmentation. Finally, these concepts were integrated into a prototype and a system for left hand fingering detection was developed.

Sight-reading is the act of performing a piece of music at first sight. This can be a difficult task to master, because it requires extensive knowledge of music theory, practice, quick thinking, and most importantly, a wide variety of musical material. A musician can only effectively sight-read with a new piece of music. This not only requires many resources, but also musical pieces that are challenging while also within a player's abilities. A Thesis presented to the Faculty of California Polytechnic State University, San Luis Obispo, and published June 2016 by Drew Schulz entitled: “ Pianote: A Sight - Reading Program That Algorithmically Generates Music Based On Human Performance ”, which is incorporated in its entirety for all purposes as if fully set forth herein, presents Piallote, a sight-reading web application for pianists that algorithmically generates music based on human performance. Pia Note's goal is to alleviate some of the hassles that pianists face when sight-reading. Pia Note presents musicians with algorithmically generated pieces, ensuring that a musician never sees the same piece of music twice. Piallote also monitors player performances in order to intelligently present music that is challenging, but within the player's abilities. As a result, Pia Note offers a sight-reading experience that is tailored to the player. On a broader level, this thesis explores different methods in effectively creating a sight-reading application. We evaluate Pia Note with a user study involving novice piano players. The players actively practice with Piallote over three fifteen-minute sessions. At the end of the study, users are asked to determine whether Piallote is an effective practice tool that improves both their confidence in sight-reading and their sight-reading abilities. Results suggest that Pia Note does improve user's sightreading confidence and abilities, but further research must be conducted to clearly validate Pia Note's effectiveness. We conclude that Piallote has potential to become an effective sight-reading application with slight improvements and further research.

A Brain Automated Chorales (BACh), an adaptive brain-computer system that dynamically increases the levels of difficulty in a musical learning task based on pianists' cognitive workload measured by functional near-infrared spectroscopy, is presented in an article in Bucknell University Conference Paper published May 2016 by Beste F. Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, Daniel Afergan, Remco Chang, and Robert J K Jacob entitled: “ Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty based on Brain State ”, which is incorporated in its entirety for all purposes as if fully set forth herein. As users' cognitive workload fell below a certain threshold, suggesting that they had mastered the material and could handle more cognitive information, BACh automatically increased the difficulty of the learning task. We found that learners played with significantly increased accuracy and speed in the brain-based adaptive task compared to our control condition. Participant feedback indicated that they felt they learned better with BACh and they liked the timings of the level changes. The underlying premise of BACh can be applied to learning situations where a task can be broken down into increasing levels of difficulty.

In consideration of the foregoing, it would be an advancement in the art to provide a method, an apparatus, or a system, for improving the convenience, the usability, the enjoyment, or the time saving of enhancing music skills or teaching of playing a musical instrument. Preferably, such methods or systems may be providing an improved, simple, flexible, adaptive, more convenient, automatic, secure, cost-effective, reliable, versatile, time-saving, easy to install, use or monitor, has a minimum part count, portable, handheld, enclosed in a small or portable housing or vehicular, minimum hardware, and/or using existing and available components, protocols, programs and applications, and providing a better user practicing and learning experience, such as interactive and fun experience with minimum frustration, for enhancing of music skills or teaching of playing a musical instrument

A method for teaching of playing of a musical instrument to a person, may be used with a client device that may comprise a microphone, a sounder (such as a speaker), and a display, may be used with a server device that may communicate over the Internet with the client device, may be used with a first database that may associate a respective skill level value to user identifiers, and may be used with a second database that may comprise sequences of musical symbols, where each of the sequences may be associated with a respective pace or tempo.

Any method herein may comprise obtaining, by the client device, an identifier of the person; sending, by the client device to the server device, the person identifier; receiving, by the server device from the client device, the person identifier; determining, by the server device by using the first database, a first skill level value associated with the received person identifier; selecting, by the server device, a first sequence of musical symbols from the second database that is associated with a first tempo or pace; sending, by the server device to the client device, the selected first sequence of musical symbols and the first pace; receiving, by the client device from the server device, the selected first sequence of musical symbols and the first pace; displaying, to the person by the display in the client device, the received first sequence of musical symbols at a second pace that is lower than the first pace and is based on the first pace and on the first skill level value; capturing, by the microphone in the client device, a sound from the musical instrument; sending, by the client device to the server device, the captured sound; receiving, by the server device from the client device, a digital representation of the captured sound; analyzing, by the server device, the captured sound and checking whether the captured sound matches the first sequence of musical symbols; determining, by the server device, the amount of musical symbols in the first sequence that do not match with the captured sound; and updating, by the server device, the skill level value associated with the person identifier in response to the amount of the musical symbols that do not match with the captured sound.

Any action or step herein, such as the displaying at the second pace comprises inducing, by the server device, by the client device, or by any combination thereof, a delay that is according to, or based on, the second pace.

Any method herein may be used with a second sequence of symbols that may be associated with the first pace and may be stored in the server device. The method may further comprise receiving, by the client device from the server device, the second sequence of musical symbols and the updated skill level value; displaying, to the person by the display in the client device, the received second sequence of musical symbols at a third pace that is based on the first pace and on the updated skill level value; capturing, by the microphone in the client device, a sound from the musical instrument; sending, by the client device to the server device, the captured sound; receiving, by the server device from the client device, a digital representation of the captured sound; analyzing, by the server device, the captured sound, and checking whether the captured sound matches the second sequence of musical symbols; determining, by the server device, the amount of musical symbols in the second sequence that do not match with the captured sound; and updating, by the server device, the skill level value associated with the person identifier in response to the amount of the musical symbols that do not match with the captured sound.

Any device herein, such as the server device, may comprise multiple parts of a musical piece, each represented by a distinct sequence of symbols Any method herein may further comprise selecting the second sequence from the multiple parts sequences. Any device herein, such as the server device, may comprise at least 2, 3, 4, 5, 7, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, or 100 parts, or may comprise less than 3, 4, 5, 7, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, or 200 parts. Any parts herein, and any two parts herein, may be sequential or non-sequential in the musical piece. Any parts herein, and any two parts herein, may be overlapping or non-overlapping.

The duration of any one of the parts herein, of each of at least two of the parts herein, or of each one of the parts herein, may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the musical piece playing duration. Alternatively or in addition, the duration of any one of the parts herein, of each of at least two of the parts herein, or of each one of the parts herein, may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the musical piece playing duration. Alternatively or in addition, the duration of any one of the parts herein, of each of at least two of the parts herein, or of each one of the parts herein, may be at least at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or 1000 seconds, or may be less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000 seconds.

The number of symbols of any one of the parts herein, of each of at least two of the parts herein, or of each one of the parts herein, may be at least may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the number of symbols relating in the entire musical piece playing duration.

Any two sequences herein, such as the first and the second sequences herein, may be part of a musical piece or any part thereof, and any second sequence herein may be played immediately after, or mot immediately after, the first sequence in the musical piece. Alternatively or in addition, any two sequences herein, such as the first and the second sequences herein, may be overlapping, partially overlapping, or non-overlapping.

Any sequences herein, such as the first and the second sequences herein, may represent the same, or different, playing time. Further, any sequences herein, such as the first and the second sequences herein, may consist of, or may comprise, the same, or different, number of symbols.

Any method herein may be used with a threshold, and may further comprise comparing the amount of the non-matched musical symbols to the threshold, and acting in response to the number of non-matched musical symbols being above the threshold. Any acting herein may comprise storing, by the server device in the first database, an identifier of the first sequence associated with the person identifier. Alternatively or in addition, any acting herein may comprise sending, by the server device to the client device, a message; and receiving, by the client device from the server device, the message, and any message herein may comprise identification of the non-matching symbols. Any method herein may further comprise notifying, such as by the client device to the person, in response to the receiving of the message, of the non-matching symbols, such as by displaying, to the person on the display, of the non-matching symbols as marked.

Any threshold herein may be at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or 1000 non-matching symbols, or may be less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000 non-matching symbols. Alternatively or in addition, any threshold herein may be at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, of the number of symbols in any sequence, such as in the first sequence, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95% of the number of symbols in any sequence, such as in the first sequence.

Any displaying herein may comprise displaying the musical symbols in the first sequence one at a time at the second pace, any capturing herein may comprise capturing the sound from the musical instrument that may be responsive to the last displayed music symbol, any analyzing herein of any captured sound and the checking may comprise analyzing of any received sound and comparing to the last displayed music symbol. Any determining herein of the amount of musical symbols may comprise counting the number of received sounds that may not match with the respective displayed symbols. Any method herein may further comprise, in response to counting of a single non-matching symbol, or in response to counting of two or more non-matching symbols, stopping of the displaying of the musical symbols in the first sequence. Alternatively or in addition, any method herein may be used with a second sequence of symbols that may be associated with the first pace and may be stored in the server device, and may further comprise in response to counting of a single non-matching symbol, or in response to counting of two or more non-matching symbols, displaying, to the person by the display in the client device, the second sequence of musical symbols at the second pace.

Alternatively or in addition, any method herein may be used with a second sequence of symbols that may be associated with the first pace and may be stored in the server device, and may further comprise in response to counting of a single non-matching symbol, or in response to counting of two or more non-matching symbols: sending, by the server device to the client device, a message; receiving, by the client device from the server device, the message; displaying, by the client device on the display, the received message; obtaining, by the client device from the person using an input component, a command; sending, by the client device to the server device, the obtained command; receiving, by the server device from the client device, the obtained command; and acting in response to the obtained command.

A method may be used with a musical piece, or part thereof, that may be cooperatively played by a first musical instrument according to first sequence of musical symbols, by a second musical instrument according to second sequence of musical symbols, and by a third musical instrument according to third sequence of musical symbols. Any method herein may be used with a first client device associated with the first musical instrument and operated by a first person that is identified by a first person identifier, with a second client device associated with the second musical instrument and operated by a second person that is identified by a second person identifier, and with a third client device associated with the third musical instrument and operated by a third person that is identified by a third person identifier, and each of the first, second, and third client devices may comprise a microphone, a sounder, and a display and communicates over the Internet with a server device that may store the first, second, and third sequences.

Any method herein may comprise (e.g., synchronously) sending, by the server device to the first, second, and third client devices, respectively the first, second, and third sequences of musical symbols; receiving, by first, second, and third client devices from the server device, respectively the first, second, and third sequences; displaying, to the first, second, and third persons by the respective display in the respective first, second, and third client devices, the respective received first, second, and third sequences; capturing, by the respective microphone in the first, second, and third client devices, respective first, second, and third sounds from the respective first, second, and third musical instruments; sending, by the first, second, and third client devices to the server device, the respective captured first, second, and third sounds; receiving, by the server device from the first, second, and third client devices, the respective captured first, second, and third sounds; analyzing, by the server device, the received captured first, second, and third sounds; sending, by the server device only to the second and third client devices, the received captured first sound; receiving, by the second and third client devices from the server device, the received captured first sound; emitting, by the sounder in the second and third client devices, the received captured first sound; sending, by the server device only to the third client device, the received captured second sound; receiving, by the third client device from the server device, the received captured second sound; and emitting, by the sounder in the third client devices, the received captured second sound.

Any method herein may further comprise sending, by the server device to the first client device, the received captured second and third sounds; receiving, by the first client devices from the server device, the received captured second and third sounds; and emitting, by the sounder in the first client device, the received captured second and third sounds. Any method herein may further comprise sending, by the server device to the second client device, the received captured third sound; receiving, by the second client device from the server device, the received captured third sound; and emitting, by the sounder in the second client device, the received captured third sound.

Any first musical instrument may consist of, or may comprise, a string instrument, and at least one of the first and second musical instruments consists of, or comprises, a string instrument, or at least one of the first and second musical instruments may consist of, or may comprise, an instrument that is not a string instrument. Alternatively or in addition, any first musical instrument may consist of, or may comprise, a woodwind instrument, and at least one of the first and second musical instruments consists of, or comprises, a woodwind instrument, or at least one of the first and second musical instruments may consist of, or may comprise, an instrument that is not a woodwind instrument.

Alternatively or in addition, any first musical instrument may consist of, or may comprise, a brass instrument, and at least one of the first and second musical instruments consists of, or comprises, a brass instrument, or at least one of the first and second musical instruments may consist of, or may comprise, an instrument that is not a brass instrument. Alternatively or in addition, any first musical instrument may consist of, or may comprise, a percussion instrument, and at least one of the first and second musical instruments consists of, or comprises, a percussion instrument, or at least one of the first and second musical instruments may consist of, or may comprise, an instrument that is not a percussion instrument.

Any one of, any of two of, each one of, any first, second, and third musical instruments herein may comprise or may consist of an instrument that may be selected from a group that consists of a soprano instrument, an alto instrument, a tenor instruments, a baritone instruments, and a bass instruments.

Any musical piece herein may be further cooperatively played by a fourth musical instrument according to a fourth sequence of musical symbols. Any method herein may further comprise sending, by the server device to the first, second, or third client device, the fourth sequence, (e.g., synchronized) with the sending of the first, second, and third sequences of musical symbols; receiving, by the respective first, second, or third client device from the server device, the fourth sequence; and emitting, by the sounder in the respective first, second, or third client device, a sound according to the fourth sequence that mimics the actual playing of the fourth musical instrument.

Any sending herein may comprise sending to the first, second, and third client devices, and any emitting herein may comprise emitting, by the sounder in each of the first, second, and third client device. Any fourth instrument herein may be is identical to, may be similar to, or may be different from, any first, second, or third musical instrument.

Any method herein may further comprise converting the musical symbols in the fourth sequence into corresponding sound data that may mimic the playing of the fourth sequence by the fourth musical instrument, and any emitting herein may comprise sounding of the converted sound data. Any converting herein may be performed by any server device herein or by any client device herein.

Any method herein may further comprise obtaining, by each of the client devices, the respective person identifier; sending, by each of the client devices to the server device, the respective person identifier; and receiving, by the server device from each of the client device, the respective person identifier. Any server device herein may further store a first skill level value associated with the first person identifier, a second skill level value associated with the second person identifier, and a third skill level value associated with the third person identifier, and any musical piece herein may be associated with a first pace.

Any (e.g., synchronously) sending of each of the first, second, and third sequences herein, or any displaying of each of the first, second, and third sequences, may be at a second pace that is equal to, or lower than, the first pace. Any second pace herein may be based on the first, second, and third skill level values, such as may be based on the lowest value of the first, second, and third skill level values.

Each of the skill level values herein may be represented by a numerical value, and any second pace herein may be calculated based on a linear or non-linear function of the numerical values multiplied by the first pace. Any (e.g., synchronously) sending herein and any displaying herein at the second pace may comprise inducing a delay that may be according to, or may be based on, the second pace. Any inducing of any delay herein may be performed by any server device or by each of the client devices.

Any method herein may further comprise checking, by any server device, whether the each of the received captured sounds matches the respective sequence; determining, by any server device, the amount of musical symbols in each of the sequences that do not match with the respective captured sound; and updating, by any server device, the skill level value associated with the respective person identifier in response to the amount of the musical symbols that do not match in the respective captured sound.

Any musical instrument herein may comprise a keyboard that may consist of a row of keys for being played by the person, and each of the musical symbols herein may be associated with one or more of the keys, and any checking herein may comprise comparing one or more symbols in the first sequence with the respective sounds when respective one or more keys are pressed by the person. Any analyzing of any captured sound herein may comprise extracting a feature of the captured sound, and any extracting of any feature herein may comprise, or may be based on, time-domain analysis, frequency-domain analysis, or time-frequency domain analysis. Any extracted feature herein may comprise, or may consist of, a tone frequency, an intensity of a tone, or a duration of a tone, a Chroma feature, zero crossings, a peak amplitude, a rise-time, an energy, or a time-delay.

Any analyzing of any captured sound herein may comprise, or may be based on, a time-domain analysis, or a frequency-domain analysis that may comprise, or may be based on, a frequency-domain representation that may be formed using, or may be based on, Fourier series, Fourier transform, Discrete Fourier Transform (DFT), Laplace transform, Z transform, or Wavelet transform. Alternatively or in addition, any feature herein may be extracted by using a Mel-Frequency Analysis, calculating Mel-Frequency Cepstral Coefficients (M FCC), using a Linear Predictive Coding (LPC), or calculating LPC coefficients.

Any analyzing or checking herein may be based on, or may use, an Artificial Neural Network (ANN) that may be trained to classify any captured sound to a musical symbol. Any ANN herein may consist of, may be based on, or may use, a Feedforward Neural Network (FNN), a Recurrent Neural Network (RNN), or a deep convolutional neural network. Further, Any ANN herein may consist of, may be based on, or may use, at least 3, 4, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers, or less than 4, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, or 50 layers.

Any ANN herein may be a Deep Neural Network (DNN), that may comprise, or may consist of, a Recurrent neural network (RNNs), a Convolutional deep Neural Network (CNNs), an AutoEncoder (AE), a Deep Belief Network (DBN), or a Restricted Boltzmann machine (RBM).

Any database herein, such as the second database, may further comprise a second sequence of musical symbols for playing of an additional musical instrument. Any method herein may further comprises sending, by the server device to the client device, the second sequence; receiving, by the client device from the server device, the second sequence; and emitting, by a sounder in the client device, a sound according to the second sequence that mimics the playing of the additional musical instrument. Any sequence herein, such as the first sequence, may be for playing of the musical instrument, and any musical piece herein or any part thereof may comprise any first and second sequences. Any emitting of any sound herein may be at any pace, such as at the second pace, and any displaying herein of the first sequence and the emitting of the sound are synchronized, such as being part of the musical piece.

Any method herein may further comprise converting, by any device such as by the server device or by the client device) any musical symbols in any sequence, such as in the second sequence, into corresponding sound data that may mimic the playing of the second sequence by the additional musical instrument, and any emitting herein may comprise sounding the converted sound data. Any method herein may further comprise selecting, by any person, any additional musical instrument, and sending, such as by the client device to the server device, the identification of the selected additional musical instrument. Any sounder herein may consist of, may comprise, may use, or may be based on, an electromagnetic loudspeaker, a piezoelectric speaker, an Electro-Static Loudspeaker (ESL), a ribbon magnetic loudspeaker, a planar magnetic loudspeaker, a bending wave loudspeaker, a piezoelectric transducer, crystal-based transducer, a capacitive transducer, or a magnetostrictive transducer.

Any method herein may be used with an additional client device that communicates over the Internet with the server device, and any method herein may further comprise sending, by the server device to the additional client device, the received captured sound.

Any method herein may be used with an additional client device that may communicate over the Internet with the server device, and may further comprise receiving, by the server device from the additional client device, a first sound; sending, by the server device to the client device, the first sound; and emitting, by a sounder in the client device, the first sound. Any sounder herein may consist of, may comprise, may use, or may be based on, an electromagnetic loudspeaker, a piezoelectric speaker, an Electro-Static Loudspeaker (ESL), a ribbon magnetic loudspeaker, a planar magnetic loudspeaker, a bending wave loudspeaker, a piezoelectric transducer, crystal-based transducer, a capacitive transducer, or a magnetostrictive transducer.

Any database herein, such as the second database may further comprise a second sequence of musical symbols for playing of an additional musical instrument, and may further comprise sending, by the server device to the additional client device, the second sequence. Any method herein may be used with a musical piece, and the first sequence may be for playing of the musical instrument, and the musical piece or a part thereof may comprise the first and second sequences. Any displaying herein of any sequence, such as the first sequence, and any emitting hereon of any sound, such as the first sound, may be synchronized.

Any method herein may further comprise obtaining, by the client device, a text that may include instructions or cues for playing on the musical instrument of the selected first sequence of musical symbols. Any text herein may comprise identification of the musical symbols in the selected first sequence of musical symbols, or may comprise instructions or cues associated with operating of the musical device. Any musical instrument herein may comprise a keyboard that may consist of a row of keys, and each of the instructions or cues herein may be associated with one or more keys.

Any method herein may further comprise converting the selected first sequence of musical symbols to generate the text, and any converting herein may be performed by the client device. Alternatively or in addition, any converting herein may be performed by the server device, and any method herein may further comprise sending the text, by the server device to the client device, and any obtaining by the client device may comprise receiving, by the client device from the server device, the text.

Any method herein may further comprise sounding, such as a human voice, by the client device, the text synchronized with the displaying of the received first sequence of musical symbols. Any sounding herein may comprise, or may be based on, a speech synthesizing or Text-to-Speech (TTS) scheme, that may use, or may be based on, a speech synthesizer that may be based on, or may use, a concatenative type, using unit selection, diphone synthesis, or domain-specific synthesis. Alternatively or in addition, any sounding herein may comprise, or may be based on, a speech synthesizing or Text-to-Speech (TTS) scheme, that may use, or may be based on, a speech synthesizer that may be based on, or may use, a formant type or may be Hidden Markov models (HMM) based. Alternatively or in addition, any sounding herein may comprise, or may be based on, a speech synthesizing or Text-to-Speech (TTS) scheme, that may use, or may be based on, stored prerecorded human voice.

Any method herein may further comprise haptic notifying, by the client device, in response to the selected first sequence of musical symbols. Any haptic notifying herein may comprise, may use, or may be based on, cutaneous, kinaesthetic, or orhaptic technology. Any haptic notifying herein may use vibration that may be produced by an Eccentric Rotating Mass (ERM) actuator, a Linear Resonant Actuator (LRA), piezoelectric actuators, an unbalanced motor, a loudspeaker, an ultrasound transducer, or an air vortex ring.

Any device herein, such as the client device, may comprise a projector, and any displaying herein may comprise illuminating, by the projector, elements of the musical instrument that may be associated with the received first sequence of musical symbols.

Any musical instrument herein may further comprise a keyboard that may consist of a row of keys, and any displaying herein may comprise illuminating, by the projector, on one or more keys that may be associated with the received first sequence of musical symbols. Any projector herein may comprise, or may consist of, an Eidophor projector, Liquid Crystal on Silicon (LCoS or LCOS) projector, LCD projector, MEMS projector, or Digital Light Processing (DLP™) projector.

Any method herein may further comprise notifying a feedback message, such as a text message, to the person in response to the amount of the musical symbols that do not match with the captured sound, or in response to updating the skill level value.

Any notification herein may comprise sounding as a human voice, by the client device, the text message, and any sounding herein may comprise, or may be based on, a speech synthesizing or Text-to-Speech (US) scheme. Any speech synthesizing or Text-to-Speech (US) herein may use, or may be based on, a speech synthesizer that may be based on, or may use, a concatenative type, using unit selection, diphone synthesis, domain-specific synthesis, a formant type or is Hidden Markov models (HMM) based. Alternatively or in addition, any notifying herein may comprise displaying, to the person by the display, the message.

Any method herein may be used with an electronic musical instrument that may comprise a port for outputting messages in response to actions of the person, and each of the musical symbols herein, such as in the selected first sequence, may correspond to a respective person action. Any method herein may comprise receiving, from the electronic musical instrument, the person actions; sending, by the client device to the server device, the received person actions; receiving, by the server device from the client device, the person actions; analyzing, by the server device, the person actions and checking whether the person actions match the first sequence of musical symbols; determining, by the server device, the amount of musical symbols in the first sequence that do not match with the person actions; and updating, by the server device, the skill level value associated with the person identifier in response to the amount of the musical symbols that do not match with the person actions.

Any receiving herein from any electronic musical instrument of the person actions may comprise receiving serial data stream according to an industry standard that may be according to, or may be based on, MIDI standard protocol. Further, any checking whether the person actions match the first sequence of musical symbols may be according to, or may be based on, MIDI standard protocol. Any electronic musical instrument herein may comprise, or may consist of, a MIDI controller or a MIDI keyboard.

Any port herein may comprise, or may consist of, a first antenna, and any receiving herein from the electronic musical instrument of the person actions may comprise receiving over a wireless communication by a second antenna using a wireless transceiver in the client device, and any wireless communication herein may comprise a Wireless Personal Area Network (WPAN), any wireless transceiver herein may comprise a WPAN transceiver, and any of the antennas herein may comprise an WPAN antenna. Any WPAN herein may be according to, may be compatible with, or may be based on, Bluetooth™, Bluetooth Low Energy (BLE), or IEEE 802.15.1-2005 standards, or wherein the WPAN is a wireless control network that may be according to, or may be based on, Zigbee™, IEEE 802.15.4-2003, or Z-Wave™ standards.

Alternatively or in addition, any port herein may comprise, or may consist of, a first connector, and any receiving herein from the electronic musical instrument of the person actions may comprise receiving over a wired point-to-point cable by a second connector in the client device. Any of the connectors herein may be a MIDI DIN connector, and any communication over the cable herein may be according to, or may be based on, MIDI standard protocol. Alternatively or in addition, any of the connectors herein may be a Universal Serial Bus (USB) connector, and any communication over any cable herein may be according to, or may be based on, USB 2.0 or USB 3.0 standard.

Any selecting herein any sequence of musical symbols, such as the first sequence of musical symbols, may comprises selecting, by the server device, a second sequence of musical symbols from the second database; and creating, by the server device, the first sequence of musical symbols from the selected second sequence of musical symbols by changing an arrangement of the selected second sequence of musical symbols for producing a simplified arrangement as the first sequence of musical symbols. Any changing herein of any arrangement to a simplified arrangement may be based on, or may be according to, a skill level value, such as the first skill level value

Any sequence of musical symbols herein, such as the first sequence of musical symbols, may be associated with a distinct complexity or difficulty related value that may be equal to, or may be based on, any skill level value, such as the first skill level value, associated with the person identifier. Any method herein may be used with multiple musical pieces, and any selecting herein, such as by the server device or by the client device, of any sequence of musical symbols from any database, such as selecting of the second sequence of musical symbols from the second database, may comprises selecting of a musical piece from the multiple musical pieces. Any sequence of musical symbols herein, such as the second sequence of musical symbols, may be a non-simplified original sequence of musical symbols of the selected musical piece.

Any method herein may further comprise, responsive to any updating of the skill level value, creating, by any device such as the server device or the client device, a third sequence of musical symbols from the selected second sequence of musical symbols, such as by changing an arrangement of the selected second sequence of musical symbols for producing a simplified arrangement as the third sequence of musical symbols.

Any changing herein of any arrangement to a simplified arrangement may be based on, or may be according to, any skill level value, such as the updated skill level value. Any sequence of musical symbols herein, such as the third sequence of musical symbols, may be associated with a distinct complexity or difficulty related value that may be equal to, or may be based on, any skill level value that may be associated with the person identifier, such as the first skill level value that is associated with the person identifier.

Any method herein may further comprise, sending, by the server device to the client device, the third sequence of musical symbols; receiving, by the client device from the server device, the third sequence of musical symbols; displaying, to the person by the display in the client device, the third sequence of musical symbols; capturing, by the microphone in the client device, a sound from the musical instrument; sending, by the client device to the server device, a digital representation of captured sound; receiving, by the server device from the client device, the digital representation of the captured sound; analyzing, by the server device, the captured sound and checking whether the captured sound matches the third sequence of musical symbols; determining, by the server device, the amount of musical symbols in the third sequence that do not match with the captured sound; and updating, by the server device, the skill level value associated with the person identifier in response to the amount of the musical symbols that do not match with the captured sound.

Any producing herein of any simplified arrangement, such as the first or third sequence of musical symbols, may comprise changing a hand or finger motion relative to a geometry of the playing mechanism of the musical instrument; changing a playing speed, a note density, a Distinct Stroke Rate (DSR), a Pitch Entropy (PE), a Hand Displacement Rate (HDR), a Hand Stretch (HS), a PolyPhony Rate (PPR), an Altered Note Rate (ANR), or a fingering complexity (FCX); or a changing fingers positioning, a transition between symbols, a tempo, or decorations. Alternatively or in addition, any producing herein of any simplified arrangement, such as the first or third sequence of musical symbols, may comprise changing the physical difficulty of playing the sequence based on the physical structure of the musical instrument; changing a rhythmic difficulty that comprises a rate of playing and the irregularity of the musical symbols in time; changing a harmonic difficulty that comprises a combination complexity of simultaneously or sequentially played musical symbols; changing an expressivity difficulty that comprises a need to control and vary the loudness and timbre of the musical symbols; or changing the number of simultaneously played symbols or chords.

Any selecting herein, such as any selecting of a musical piece or a part thereof, may be based on, or may use, a First-In-First-Out (FIFO) scheme, a Last-In-First-Out (LIFO) scheme, a sequential or cyclic selection, a random selection, or any combination thereof. Alternatively or in addition, any selecting herein may be based on, or may use, a random selection that may be based on, or may use, one or more random numbers generated by a random number generator. Any random number generator herein may be hardware based, and may be based on, or may use, thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena. Alternatively or in addition, any random number generator may be software based, such as based on executing an algorithm for generating pseudo-random numbers.

Any method herein may be used with a plurality of musical pieces, and any database herein, such as the second database, may comprise multiple sequences of musical symbols for each of the plurality of musical pieces, and a distinct complexity or difficulty related value may be associated with each of the multiple sequences of musical symbols of each of the plurality of musical pieces. Any selecting herein may comprise selecting of a musical piece from the plurality of musical pieces, and may further comprise selecting of a first sequence from the multiple sequences of musical symbols of the selected musical piece. Each of the plurality of musical pieces may be associated with an associated sequence of musical symbols that may be an original non-simplified version of the respective musical piece.

Any database herein, such as the second database, may further comprise a distinct complexity or difficulty related value associated with each of the multiple sequences of musical symbols, and any selecting herein may be based on comparing the complexity or difficulty values with the first skill level value associated with the person identifier. Each of the skill level values herein may be a numerical value, and each of the complexity or difficulty values may be a numerical value, and any selecting herein may be based on, or may use, selecting of a sequence of musical symbols that may be associated with a complexity or difficulty value that may be equal to, or lower than, the first skill level value associated with the person identifier.

Any method herein may further comprise estimating or calculating the complexity or difficulty related value that may be associated with at least one of, or all of, the multiple sequences of musical symbols. Any estimating or calculating of the complexity or difficulty related value herein may be performed by the server device, by the client device, or any combination thereof. Further, any estimating or calculating of the complexity or difficulty related value herein may be based on estimated complexity or difficulty of playing the at least one of the multiple sequences of musical symbols on the musical instrument.

Any method herein may further comprise, responsive to any updating the skill level value, selecting, such as based on comparing the complexity or difficulty values with the updated skill level value, a second sequence of musical symbols from the multiple sequences of musical symbols. Any two sequences herein, such as the first and second sequences, may be representations of the same musical piece, and the second sequence may follow the first sequence when playing the musical piece. Any method herein may further comprise sending, by the server device to the client device, the selected second sequence of musical symbols; receiving, by the client device from the server device, the selected second sequence of musical symbols; and displaying, to the person by the display in the client device, the received second sequence of musical symbols.

Any estimating or calculating herein of the complexity or difficulty related value of the at least one of the multiple sequences of musical symbols may be based on a required hand or finger motion and the geometry of the playing mechanism of the musical instrument, a playing speed, a note density, a pitch entropy, a hand displacement rate, a hand stretch, or fingering complexity.

Alternatively or in addition, any estimating or calculating herein of the complexity or difficulty related value of the at least one of the multiple sequences of musical symbols may be based on, or may comprise, associating identifying phrases in the sequence with a weighted importance value; estimating or calculating a distance of notes, tempo, or ease of interpretation; estimating or calculating fingers positioning, transition between symbols, tempo, or decorations; estimating or calculating motoric difficulty that comprises the physical difficulty of playing the sequence based on the physical structure of the musical instrument; rhythmic difficulty that comprises a rate of playing and the irregularity of the musical symbols in time; harmonic difficulty that comprises a combination complexity of simultaneously or sequentially played musical symbols; or expressivity difficulty that comprises a need to control and vary the loudness and timbre of the musical symbols.

Alternatively or in addition, any estimating or calculating herein of the complexity or difficulty related value of the at least one of the multiple sequences of musical symbols may be based on, or may comprise, extracting a feature of the sequence of musical symbols, and any extracting of any feature herein may comprise, or may be based on, time domain analysis, frequency domain analysis, or time-frequency domain analysis, using a Mel-Frequency Analysis, calculating Mel-Frequency Cepstral Coefficients (MFCC), using a Linear Predictive Coding (LPC), or calculating LPC coefficients. Any extracted feature herein may comprise, or may consist of, a tone frequency, an intensity of a tone, a duration of a tone, a Chroma feature, zero crossings, a peak amplitude, a rise-time, an energy, a time-delay, a frequency-domain representation that is formed using, or based on, Fourier series, Fourier transform, Discrete Fourier Transform (DFT), Laplace transform, Z transform, Wavelet transform,

Any sequence of musical symbols herein may be based on, may consist of, or may comprise, instructions for playing a musical piece or a part thereof, and any pace herein may be based on, may consist of, or may comprise, a tempo that may be associated with the musical piece. Any tempo or pace herein may be based on, may consist of, or may comprise, Larghissimo, Adagissimo, Grave, Largo, Lento, Larghetto, Adagio, Adagietto, Andante, Andantino, Marcia moderato, Andante moderato, Moderato, Allegretto, Allegro moderato, Allegro, Molto Allegro, Vivace, Vivacissimo, Allegrissimo, Allegro vivace, Presto, or Prestissimo. Alternatively or in addition, any tempo herein may be above 24 beats per minute (bpm), may be above 200 bpm, or may be within one of the ranges of 25-45 bpm, 40-60 bpm, 45-60 bpm, 60-66 bpm, 66-76 bpm, 72-76 bpm, 70-80 bpm, 76-108 bpm, 80-108 bpm, 83-85 bpm, 92-98 bpm, 98-112 bpm, 102-110 bpm, 116-120 bpm, 120-156 bpm, 124-156 bpm, 156-176 bpm, 172-176 bpm, or 168-200 bpm.

Any pace or tempo herein that is based on another tempo or pace, may be less than 95%, 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, or 10% of the another pace or tempo. Alternatively or in addition, any pace or tempo herein that is based on another tempo or pace, may be more than 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, or 5% of the another pace or tempo. Alternatively or in addition, any pace or tempo value herein is calculated based on a linear or non-linear function of a numerical value multiplied by another pace or tempo value, and the numerical value may be a function of, or a representation of, a skill level such as the first skill level.

Any musical instrument herein may consists of, or may comprise, a Soprano instrument such as a flute, a violin, a soprano saxophone, a trumpet, a clarinet, an oboe, or a piccolo. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, an Alto instrument, such as an alto saxophone, an French horn, an English horn, a viola, or an alto horn. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, a Tenor instrument, such as a trombone, a tenoroon, a tenor saxophone, a tenor violin, a guitar, or a tenor drum. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, a Baritone instrument, such as a bassoon, a baritone a saxophone, a bass clarinet, a cello, a baritone horn, or an euphonium. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, a Bass instrument, such as a double bass, a bass guitar, a contrabassoon, a bass saxophone, a tuba, or a bass drum.

Any musical instrument herein may consist of, or may comprise, a string instrument that produces sound by means of vibrating strings, such as a guitar, an electric bass, a violin, a viola, a cello, a double bass, a banjo, a mandolin, an ukulele, or a harp. Any string instrument herein may consist of, or may comprise, a lute instrument in which the strings are supported by a neck and a bout, a harp instrument, in which the strings are contained within a frame, or a zither instrument, in which the strings are mounted on a body. Further, any string instrument herein be configured to be played by plucking, bowing, or striking. Furthermore, the string instrument any string instrument herein may be an acoustic instrument, or alternatively may comprise an electric amplification.

Any musical instrument herein may consist of, or may comprise, a woodwind instrument that produces sound when the player blows air against a sharp edge or through a reed, causing the air within its resonator to vibrate, such as a flute, a piccolo, an oboe, an English horn, a clarinet, a bass clarinet, a bassoon, a contrabassoon, or a saxophone. Any woodwind instrument herein may consist of, or may comprise, a flute that is configured to produce sound when air is blown across an edge, and any flute herein may consist of, or may comprise, an open flute, in which the player's lips form a stream of air which goes directly from the players lips to the edge, or a closed flute, in which in which the musical instrument has a channel to form and direct the air stream over an edge. Alternatively or in addition, any woodwind instrument herein may consist of, or may comprise, a reed instrument that is configured to produce sound by focusing air into a mouthpiece which then causes a reed, or reeds, to vibrate, and any reed instrument herein may consist of, or may comprise, a single reed instrument that uses a reed, which is a thin-cut piece of cane or plastic that is held against the aperture of a mouthpiece with a ligature, so that when air is forced between the reed and the mouthpiece, the reed vibrates to create a sound, a double reed instrument that uses two precisely cut small pieces of cane that are joined together at the base and are inserted into the top of the instrument to vibrate as air is forced between the two pieces, or a capped double reed instrument configured to vibrate when the player blows through a hole in a cap that covers the reed.

Any musical instrument herein may consist of, or may comprise, a brass instrument that produces sound by sympathetic vibration of air in a tubular resonator in sympathy with the vibration of the player's lips, such as a Trumpet, a Cornet, a Horn, a Trombone, a Saxhorn, or a Tuba. Any brass instrument herein may consist of, or may comprise, a valved brass instrument that uses a set of valves operated by the player's fingers that introduce additional tubing, or crooks, into the instrument for changing its overall length, or a slide brass instrument that uses a slide to change the length of tubing.

Any musical instrument herein may consist of, or may comprise, a percussion instrument that is configured to produce sound by being struck or scraped by a beater, such as a timpani, a snare drum, a bass drum, cymbals, a triangle, a tambourine, a glockenspiel, or a xylophone. Any percussion instrument herein may consist of, or may comprise, a pitched percussion instrument that produces notes with an identifiable pitch, or an unpitched percussion instrument that produces notes or sounds in an indefinite pitch.

Any musical instrument herein may comprise, or may be based on, an acoustic, stringed musical instrument, configured so that the strings are struck by wooden hammers that are coated with a softer material, and the musical instrument may further comprise a keyboard that may consist of a row of keys, may be configured so that the performer presses down or strikes with the fingers and thumbs of both hands to cause the hammers to strike the strings. Any musical instrument herein may comprise, or may consist of, a piano, that may comprise, or may consist of, a grand piano, an upright piano, or an electronic piano.

Any playing of any musical instrument herein may consists of, may comprises, or may be supplemented with, a vocal music, that may be with or without instrumental accompaniment, and may use lyrics, such as singing.

Any sequence herein may include musical symbols, and each of the musical symbols in the sequence may be according to a musical notation convention. Further, any sequence herein may be part of, or may consist of, a music sheet. Any sequence herein may include one or more clefs that may define the pitch range or the tessitura of the symbol on which it is placed, one or more musical notes that may denote a musical sound or the pitch and duration of a musical sound, one or more accidentals that may denote a pitch or a pitch class, one or more key signatures that may define the prevailing key of the music that follows, one or more time signatures that may define the meter of the music, one or more lines or a note relationships, one or more dynamics that may indicate the relative intensity or volume of a musical line, one or more articulations that may specify how to perform individual notes within a phrase or passage, one or more ornaments that may modify the pitch pattern of an individual note, one or more octave signs, repetitions or codas, one or more pedal marks, or any combination thereof.

Any method herein may be used with a threshold, and any updating herein may comprise raising or lowering of the skill level value associated with the person identifier in response to the amount of the non-matching musical symbols being less or more than the threshold. Any threshold herein may be at least 1, 2, 3, 5, 8, 10, 15, 20, 30, or 50 non-matching symbols, or may be less than 1, 2, 3, 5, 8, 10, 15, 20, 30, or 50 non-matching symbols. Alternatively or in addition, any threshold herein may be at least 0.1%, 0.2%, 0.5%, 0.8%, 1%, 2%, 3%, 5%, 8%, 10%, 15%, 20%, 30%, or 50% non-matching symbols out of the total number of symbols in the sequence, or may be less than 0.1%, 0.2%, 0.5%, 0.8%, 1%, 2%, 3%, 5%, 8%, 10%, 15%, 20%, 30%, or 50% non-matching symbols out of the total number of symbols in the sequence. Alternatively or in addition, any method herein may be used with low and high thresholds, and any updating herein may comprise unchanging the skill level value associated with the person identifier in response to the amount of the non-matching musical symbols being more than the low threshold and less than the high threshold.

Any sequence herein of any musical symbols may represents an entire of, or part of, a musical piece. Any musical piece herein may include symbols for multiple musical instruments, and any sequence herein may represent or include symbols adapted only for one specific musical instrument. Any musical piece herein may comprise, or may consist of a song, a vocal music work, or a classical music work, such as an Aria, an Cadenza, an Concerto, an Movement, an Overture, an Opera, an Sonata, an Chamber music, or an Symphony. Alternatively or in addition, any musical piece herein may comprise, or may consist of, a popular music work, such as a song, a dance, or a Funk, Country, Latin, Reggae, Hip-hop, or Polka music genre.

Any sequence of the musical symbols herein may represent a part of the musical piece that may be associated with a duration of playing the entire musical piece, or that may define a number of symbols. Any duration of any part herein is at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, or may be less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95%, of the duration of the playing time of the musical piece. Alternatively or in addition, any duration of playing any sequence herein is at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or may be less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000 seconds. Alternatively or in addition, any sequence herein may comprise, or may consist of, at least 0.1%, 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, or 90%, or may comprise, or may consist of, less than 0.2%, 0.3%, 0.5%, 0.8%, 1%, 1.5%, 2%, 3%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 50%, 60%, 70%, 80%, 90%, or 95%, of the number of symbols of the entire musical piece. Alternatively or in addition, any sequence herein may comprise, or may consist of, at least 1, 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, or 1000 symbols, or less than 2, 3, 5, 8, 10, 12, 15, 20, 25, 30, 40, 50, 60, 80, 100, 150, 200, 300, 500, 1000 or 2000 symbols.

Any microphone herein may be configured to respond to audible or inaudible sound, and may be an omnidirectional, unidirectional, or bidirectional microphone. Any microphone herein may be based on the sensing the incident sound-based motion of a diaphragm or a ribbon. Alternatively or in addition, any microphone herein may consist of, or may comprise, a condenser, an electret, a dynamic, a ribbon, a carbon, an optical microphone, or a piezoelectric microphone. Further, any microphone herein may consist of, may comprise, or may be based on, a microphone array for improving directivity.

Any display or any display screen herein may consist of, or may comprise, a monochrome, grayscale or color display and consists of an array of light emitters or light reflectors, or a projector that is based on an Eidophor, Liquid Crystal on Silicon (LCoS or LCOS), LCD, MEMS or Digital Light Processing (DLP™) technology. Any projector herein may consist of, or may comprise, a virtual retinal display. Further, any display or any display screen herein may consist of, or may comprise, a 2D or 3D video display that may support Standard-Definition (SD) or High-Definition (HD) standards, and may be capable of scrolling, static, bold or flashing the presented information.

Alternatively or in addition, any display or any display screen herein may consist of, or may comprise, an analog display having an analog input interface supporting NTSC, PAL or SECAM formats, and the analog input interface may include RGB, VGA (Video Graphics Array), SVGA (Super Video Graphics Array), SCART or S-video interface. Alternatively or in addition, any display or any display screen herein may consist of, or may comprise, a digital display having a digital input interface that may include IEEE1394, FireWire™, USB, SDI (Serial Digital Interface), HDMI (High-Definition Multimedia Interface), DVI (Digital Visual Interface), UDI (Unified Display Interface), DisplayPort, Digital Component Video or DVB (Digital Video Broadcast) interface. Alternatively or in addition, any display or any display screen herein may consist of, or may comprise, a Cathode-Ray Tube (CRT), a Field Emission Display (FED), an Electroluminescent Display (ELD), a Vacuum Fluorescent Display (VFD), or an Organic Light-Emitting Diode (OLED) display, a passive-matrix (PMOLED) display, an active-matrix OLEDs (AMOLED) display, a Liquid Crystal Display (LCD) display, a Thin Film Transistor (TFT) display, an LED-backlit LCD display, or an Electronic Paper Display (EPD) display that may be based on Gyricon technology, Electro-Wetting Display (EWD), or Electrofluidic display technology. Alternatively or in addition, any display or any display screen herein may consist of, or may comprise, a laser video display that is based on a Vertical-External-Cavity Surface-Emitting-Laser (VECSEL) or a Vertical-Cavity Surface-Emitting Laser (VCSEL). Further, any display or any display screen herein may consist of, or may comprise, a segment display based on a seven-segment display, a fourteen-segment display, a sixteen-segment display, or a dot matrix display, and may be operative to display digits, alphanumeric characters, words, characters, arrows, symbols, ASCII, non-ASCII characters, or any combination thereof.

Any sound herein may be audible or inaudible (or both), and may be omnidirectional, unidirectional, bidirectional, or provide other directionality or polar patterns. Any sounder herein may be an electromagnetic loudspeaker, a piezoelectric speaker, an Electro-Static Loudspeaker (ESL), a ribbon or planar magnetic loudspeaker, or a bending wave loudspeaker. Any sounder herein may consist of, may comprise, may use, or may be based on, an electric sound source that may convert electrical energy into sound waves, and the electric sound source may be configured to emit an audible or inaudible sound using omnidirectional, unidirectional, or bidirectional pattern. Further, any sounder herein may convert electrical energy to sound waves transmitted through the air, an elastic solid material, or a liquid, usually by means of a vibrating or moving ribbon or diaphragm.

Further, the electric sound source may consist of, may comprise, may use, or may be based on, an electromagnetic loudspeaker, a piezoelectric speaker, an electrostatic loudspeaker (ESL), a ribbon magnetic loudspeaker, a planar magnetic loudspeaker, or a bending wave loudspeaker. Alternatively or in addition, the electric sound source may consist of, may comprise, may use, or may be based on, an electromechanical scheme or a ceramic-based piezoelectric effect. Alternatively or in addition, the electric sound source may consist of, may comprise, may use, or may be based on, an ultrasonic transducer that may be a piezoelectric transducer, crystal-based transducer, a capacitive transducer, or a magnetostrictive transducer.

Any server herein may be storing, operating, or using, a server operating system, which may consist of, may comprise, or may be based on, one out of Microsoft Windows Server®, Linux, or UNIX, or may consist of, may comprise, or may be based on, one out of Microsoft Windows Server® 2003 R2, 2008, 2008 R2, 2012, or 2012 R2 variant, Linux™ or GNU/Linux-based Debian GNU/Linux, Debian GNU/kFreeBSD, Debian GNU/Hurd, Fedora™, Gentoo™, Linspire™, Mandriva, Red Hat® Linux, SuSE, and Ubuntu®, UNIX® variant Solaris™, AIX®, Mac™ OS X, FreeBSD®, OpenBSD, and NetBSD®.

Any software or firmware herein may comprise an operating system that may be a mobile operating system. The mobile operating system may consist of, may comprise, may be according to, or may be based on, Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean), Android version 4.4 (KitKat), Android version 6.0 (Marshmallow), Android version 7.0 (Nougat), Android version 8.0 (Oreo), Android version 9.0 (Pie), Android 10, Android 11, Apple iOS version 3, Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple iOS version 7, Apple iOS version 8, Apple iOS version 9, Apple iOS version 10, Apple iOS version 11, Apple iOS version 12, Apple iOS version 13, Apple iOS version 14, Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8, Microsoft Windows® Phone version 9, or Blackberry® operating system. Any Operating System (OS) herein, such as any server or client operating system, may consists of, include, or be based on a real-time operating system (RTOS), such as FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (μC/OS).

Any device herein may consist of, may comprise, may be part of, or may integrated with, a client device that may comprise a single enclosure that may house the microphone and the display. Any device herein may consist of, may comprise, may be part of, or may integrated with, a client device, that may consist of, may comprise, may be part of, or may integrated with, a hand-held enclosure, a portable enclosure, or a surface mountable enclosure. Further, any client device herein may consist of, may comprise, may be part of, or may be integrated with, a notebook computer, a laptop computer, a media player, a cellular telephone, a tablet device, or a smartphone, which may consist of, may comprise, or may be based on, an Apple iPhone 12 or a Samsung Galaxy S20.

Any device herein, such as any client device herein, may be configured or shaped to be wearable on a person, such as on an organ of the person head, and the organ may be an eye, ear, face, cheek, nose, mouth, lip, forehead, or chin. Any device herein may include an enclosure that may be constructed to have a form substantially similar to, may be constructed to have a shape allowing mounting or wearing identical or similar to, or may be constructed to have a form to at least in part substitute for, headwear, eyewear, or earpiece.

Any headwear herein may consist of, may be structured as, or may comprise, a bonnet, a cap, a crown, a fillet, a hair cover, a hat, a helmet, a hood, a mask, a turban, a veil, or a wig. Any eyewear herein may consist of, may be structured as, or may comprise, glasses, sunglasses, a contact lens, a blindfold, or a goggle. Any earpiece herein may consist of, may be structured as, or may comprise, a hearing aid, a headphone, a headset, or an earplug. Any enclosure herein may be permanently or releseably attachable to, or is part of, a clothing piece of a person, and any attaching herein may use taping, gluing, pinning, enclosing, encapsulating, a pin, or a latch and hook clip. Any clothing piece herein may be a top, bottom, or full-body underwear, or a headwear, a footwear, an accessory, an outwear, a suit, a dress, a skirt, or a top. Any device herein may comprise an annular member defining an aperture therethrough that is sized for receipt therein of a part of a human body. Any human body part herein may be part of a human hand that may consist of, or may comprise, an upper arm, elbow, forearm, wrist, or a finger. Alternatively or in addition, any human body part herein may be part of a human head or neck that may consist of, or may comprise, a forehead, ear, skull, or face. Alternatively or in addition, any human body part herein may be a part of a human thorax or abdomen that may consist of, or may comprise, a waist or hip. Alternatively or in addition, any human body part herein may be part of a human leg or foot that may consist of, or may comprise, a thigh, calf, ankle, instep, knee, or toe.

Any method herein may be in combination with a Virtual Reality (VR) system that simulates a virtual environment to the person, and any communication herein with the VR system may be wired or wireless. Any VR system herein may comprises a Head-Mounted Display (HMD), and any device herein, such as any client device herein, may comprise, may be part of, may consist of, or may be integrated with, the HMD. Any method herein may further be used in an Augmented Reality (AR) system or a Spatial Augmented Reality (SAR), that simulates a virtual environment to the person, and any display herein may comprise, may be implemented as, or may be integrated with, an Head-Mounted Display (HMD), a Head-Up Display (HUD), contact lenses, a Virtual Retinal Display (VRD).

Any integration herein, such as integration of any client device herein with any musical instrument herein, may involve sharing a component, housing in same enclosure, sharing same processor, mounting onto same surface, or sharing a same connector, which may be a power connector for connecting to a power source. Alternatively or in addition, the integration may involve sharing the same connector for being powered from same power source, or the integration may involve sharing same power supply.

Any communication herein, such as between a client device and a server device, may be over the Internet. Alternatively or in addition, the communication may be over the Internet via a wireless network, and the client device may comprise an antenna and a wireless transceiver coupled to the antenna for communication via the wireless network.

Any wireless network herein may comprise a Wireless Wide Area Network (WWAN), any wireless transceiver herein may comprise a WWAN transceiver, and any antenna herein may comprise a WWAN antenna. Any WWAN herein may be a wireless broadband network. The WWAN may be a WiMAX network, the antenna may be a WiMAX antenna and the wireless transceiver may be a WiMAX modem, and the WiMAX network may be according to, compatible with, or based on, IEEE 802.16-2009. Alternatively or in addition, the WWAN may be a cellular telephone network, the antenna may be a cellular antenna, and the wireless transceiver may be a cellular modem, where the cellular telephone network may be a Third Generation (3G) network that may use a protocol selected from the group consisting of UMTS W-CDMA, UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, and GSM EDGE-Evolution, or the cellular telephone network may use a protocol selected from the group consisting of a Fourth Generation (4G) network that use HSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be based on IEEE 802.20-2008.

Any wireless network herein may comprise a Wireless Personal Area Network (WPAN), the wireless transceiver may comprise a WPAN transceiver, and the antenna may comprise a WPAN antenna. The WPAN may be according to, compatible with, or based on, Bluetooth™, Bluetooth Low Energy (BLE), or IEEE 802.15.1-2005 standards, or the WPAN may be a wireless control network that may be according to, or may be based on, Zigbee™, IEEE 802.15.4-2003, or Z-Wave™ standards. Any wireless network herein may comprise a Wireless Local Area Network (WLAN), \ the wireless transceiver may comprise a WLAN transceiver, and the antenna may comprise an WLAN antenna. The WLAN may be according to, may be compatible with, or may be based on, a standard selected from the group consisting of IEEE 802.11-2012, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and IEEE 802.11ac. Any wireless network herein may be over a licensed or unlicensed radio frequency band that may be an Industrial, Scientific and Medical (ISM) radio band.

Any method herein may be performed partly or in full by any device, such as any client device or any server device herein. Alternatively or in addition, any method herein may be split between two or more devices, such as between a client device and a server device. Any database herein may be a relational database system, and may be Structured Query Language (SQL) based.

Any method herein may comprise communicating, such as by the device, with an Internet-connected server. Any server herein may comprise a database, and any method herein may further comprise communicating with the server for using or accessing the database. Any communicating herein with the server may be via a wireless network that may use a wireless transceiver and an antenna in the device. Any method herein may further comprise sending, via the wireless network to the server, the identified items or the estimated current location of the device.

Any method herein may use, or may be used with, a virtualization, and any communication herein, such as between the device and the server, may be executed as a virtualized network as part of a Virtual Machine (VM). Any method herein may use, or may be used with, a host computer that may implement the VM, and any method herein may further comprise executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM), and any virtualized network herein may use or may interface virtual hardware, and the virtualization may include, may be based on, or may use, full virtualization, para-virtualization, or hardware assisted virtualization.

Any network herein may be a wireless network, the first port may be an antenna for transmitting and receiving first Radio-Frequency (RF) signals over the air, and the first transceiver may be a wireless transceiver coupled to the antenna for wirelessly transmitting and receiving first data over the air using the wireless network. Alternatively or in addition, the network may be a wired network, the first port may be a connector for connecting to the network medium, and the first transceiver may be a wired transceiver coupled to the connector for transmitting and receiving first data over the wireless medium.

Any system, device, module, or circuit herein may be addressable in a wireless network (such as the Internet) using a digital address that may be a MAC layer address that may be MAC-48, EUI-48, or EUI-64 address type, or may be a layer 3 address and may be a static or dynamic IP address that may be of IPv4 or IPv6 type address. Any system, device, or module herein may be further configured as a wireless repeater, such as a WPAN, WLAN, or a WWAN repeater.

Any method herein, any step herein, any flow-chart herein, or any part thereof, may be used with a virtualization, and at least one of the steps or methods herein may be executed as part of a virtualized application as part of a Virtual Machine (VM). Any device herein, such as the analyzer device, the first device, or any part thereof, may be implemented as virtual hardware. Any virtualization herein may be used with an host computer that implement the VM, and may further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM). Any virtualized application herein or any or hardware virtualization herein may use or may interface virtual hardware. Any virtualization herein may include, may be based on, or may use, full virtualization, para-virtualization, or hardware assisted virtualization.

Any operating system herein may be used with a virtualization, and any operating system herein may be executed as a guest operating system as part of a Virtual Machine (VM). The virtualization may be implemented by a host computer that may implement the VM, and any method herein may further comprise executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM), and the guest operating system may use or interface virtual hardware. Any such virtualization herein may include, may be based on, or may use, full virtualization, para-virtualization, or hardware assisted virtualization.

Any element or entity herein may be implemented as virtualized entity. Any virtualization may include, may be based on, or may use, desktop virtualization, network virtualization, storage virtualization, application virtualization, server virtualization, or any combination thereof. Further, any virtualization herein may include, may be based on, or may use, full virtualization, para-virtualization, or hardware assisted virtualization. Further, any virtualization herein may include, may be based on, or may use, a Virtual Machine (VM) on a host computer that executes a hypervisor or Virtual Machine Monitor (VMM), and the operating system may be a guest operating system that may use or interface a virtual hardware.

Any method herein may be used with a virtualization, where at least one of the steps may be executed as part of a virtualized application as part of a Virtual Machine (VM). Alternatively or in addition, any device or server herein may be implemented as virtual hardware. Further, any method herein may be used with a host computer that may implement the VM, and any method herein may further comprise executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM), and any virtualized application herein or any hardware herein may use or may interface virtual hardware. Any virtualization herein may include, may be based on, or may uses, full virtualization, para-virtualization, or hardware assisted virtualization. At least two devices may be virtualized by the same host computer that implements the VM.

Any selection herein, such as any selection from any list or group, may be based on, or may use, a load balancing, a First-In-First-Out (FIFO) scheme, a Last-In-First-Out (LIFO) scheme, a sequential or cyclic selection, a random selection, or any combination thereof. Any random selection herein may use, or may be based on, one or more random numbers generated by a random number generator. Any random number generator herein may be hardware based, and may be using thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena. Alternatively or in addition, any random number generator herein may be software based, and may be based on executing an algorithm for generating pseudo-random numbers.

The above summary is not an exhaustive list of all aspects of the systems and methods described. Indeed, it is contemplated that this document covers all systems and methods that can be practiced from all suitable combinations and derivatives of the various aspects summarized above, as well as those disclosed in the detailed description below, and particularly pointed out in the claims filed with the application. Such combinations have particular advantages not specifically recited in the above summary.

Various aspects of the system and method are herein described, by way of non-limiting examples only, with reference to the accompanying drawings, wherein like designations denote like elements. Understanding that these drawings only provide information concerning typical embodiments of the invention and are not therefore to be considered limiting in scope:

FIG. 1 illustrates schematically a block diagram of a prior-art computer connected to the Internet.

FIG. 1 a illustrates schematically prior-art servers, clients, and a computer workstation connected via the Internet.

FIG. 2 illustrates schematically a prior-art arrangement of virtualization.

FIG. 2 a illustrates schematically a prior-art arrangement of hosted architecture of virtualization.

FIG. 2 b illustrates schematically a prior-art arrangement of bare-metal (hypervisor) architecture of virtualization.

FIG. 3 illustrates schematically a block diagram of an arrangement that includes a server that communicates over the Internet with a client device such as a smartphone or tablet.

FIG. 4 schematically illustrates a schematic diagram of an example of a feed-forward Artificial Neural Network (ANN).

FIG. 4 a schematically illustrates a schematic diagram of examples of a Deep Neural Network (DNN).

FIG. 4 b depicts schematically an HMD with and without antennas.

FIG. 4 c depicts schematically a person wearing an HMD.

FIG. 5 depicts schematically general views of various string instruments.

FIG. 5 a depicts schematically general views of various woodwind instruments.

FIG. 5 b depicts schematically general views of various brass instruments.

FIG. 5 c depicts schematically general views of various percussion instruments.

FIG. 6 depicts schematically examples of popular musical symbols according to a common music notation convention.

FIG. 6 a depicts schematically a correspondence of musical symbols and the associated piano keys.

FIG. 6 b depicts schematically a sheet music of a popular song.

FIG. 6 c depicts schematically an arrangement of a prior art online piano playing learning session using a tablet.

FIG. 7 illustrates a simplified schematic flow chart of a method for estimating and modifying complexity or difficulty of a sheet music.

FIG. 8 depicts schematically an arrangement of an online piano playing learning using a client device and a server device.

FIG. 8 a depicts schematically an arrangement of an online piano playing learning with the data paths between the client device and the server device.

FIG. 8 b depicts schematically an arrangement of an online piano playing learning with databases that relate to users and to musical pieces.

FIG. 9 illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning based on a delay in the server device.

FIG. 10 illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning based on a delay in the client device.

FIG. 11 illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning based on sequentially displaying and playing parts of a musical piece.

FIG. 11 a illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning based on sequentially displaying playing parts of a musical piece and on audible instructions or cues.

FIG. 12 illustrates a simplified schematic flow chart of a method for handling errors as part of an adaptive and interactive music learning.

FIG. 13 illustrates a simplified schematic flow chart of a method for counting errors and acting according to the count obtained during practicing of a musical piece as part of an adaptive and interactive music learning.

FIG. 13 a illustrates a simplified schematic flow chart of a method for counting errors and acting according to the count obtained during practicing of part of a musical piece as part of an adaptive and interactive music learning.

FIG. 13 b illustrates a simplified schematic flow chart of a method for no delay practicing and for counting errors and acting according to the multiple complexity- or features-based count obtained during practicing of part of a musical piece as part of an adaptive and interactive music learning.

FIG. 14 depicts schematically musical pieces database that includes multiple versions or variants for one or more of the musical pieces that corresponds to various complexity or difficulty levels.

FIG. 14 a illustrates a simplified schematic flow chart of a method for adapting a simplified version of a selected musical piece to a person skill level.

FIG. 14 b illustrates a simplified schematic flow chart of a method for adapting a simplified version of part of a selected musical piece to a person skill level during playing using stored simplified version.

FIG. 14 c illustrates a simplified schematic flow chart of a method for creating a simplified version of a selected musical piece according to a person skill level in real-time during the playing session.

FIG. 14 d illustrates a simplified schematic flow chart of a method for creating a simplified version of a selected part of a selected musical piece according to a person skill level in real-time during the playing session.

FIG. 15 depicts schematically an arrangement of an online piano playing learning using a client device.

FIG. 15 a illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning based on sequentially playing parts of a musical piece using a client device.

FIG. 15 b depicts schematically an arrangement of a MIDI controller wiredly connected to a client device.

FIG. 15 c depicts schematically an arrangement of a MIDI controller wirelessly connected to a client device that is connected for cloud-based service.

FIG. 15 d illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning using a MIDI controller based on a delay in the server device.

FIG. 16 illustrates a simplified schematic table that associates to a musical piece various musical instruments and instructions for playing the musical piece.

FIG. 16 a illustrates a simplified schematic practice tables that associate, for each user, a source of each musical instrument that is part of the musical piece.

FIG. 17 illustrates a simplified schematic flow chart of a method for cooperatively practicing a musical piece by a musical instrument while hearing a background of additional musical instruments, where at least one of the additional musical instruments is artificially produced based on the vocalizing musical symbols.

FIG. 17 a illustrates a simplified schematic flow chart of a method for cooperatively practicing a musical piece by a musical instrument while hearing a background of additional musical instruments, where at least one of the additional musical instruments is artificially produced based on stored or received sound files.

FIG. 18 depicts schematically an arrangement of three users practicing a musical piece at a same location using three different musical instruments.

FIG. 18 a illustrates a simplified schematic practice tables that associate, for each user, a source of each musical instrument that is part of the musical piece in a scenario of playing at the same location.

FIG. 19 depicts schematically an arrangement of three users practicing a musical piece at three different locations using three different musical instruments.

FIG. 19 a illustrates a simplified schematic practice tables that associate, for each user, a source of each musical instrument that is part of the musical piece in a scenario of playing at different locations, without cooperation between the users.

FIG. 20 depicts schematically an arrangement of three users practicing a musical piece at three different locations using three different musical instruments and using artificially produced sound of various musical instruments.

FIG. 20 a illustrates a simplified schematic practice tables that associate, for each user, a source of each musical instrument that is part of the musical piece in a scenario of playing at different locations, while using artificially produced sound of various musical instruments.

FIG. 21 depicts schematically an arrangement of three users practicing a musical piece at three different locations using three different musical instruments, where two users hear the playing of other musical instruments by other users.

FIG. 21 a illustrates a simplified schematic practice tables that associate, for each user, a source of each musical instrument that is part of the musical piece in a scenario of playing at different locations, where two users hear the playing of other musical instruments by other users.

FIG. 22 depicts schematically an arrangement of three users practicing a musical piece at three different locations using three different musical instruments, where all the users hear the playing of other musical instruments by all other users.

FIG. 22 a illustrates a simplified schematic practice tables that associate, for each user, a source of each musical instrument that is part of the musical piece in a scenario of playing at different locations, where all the users hear the playing of other musical instruments by all other users.

FIG. 23 depicts schematically an example of an architecture of software modules for teaching of practicing of a musical instrument that adapts to the user skill level;

FIG. 24 illustrates a simplified schematic flow chart of a method for adaptive and interactive music learning based on sequentially playing parts of a musical piece using a client device overlayed with example of an architecture of associated software modules.

FIG. 25 is a schematic illustration of a user and the entities involved in a music learning system, in accordance with some embodiments of the disclosure.

FIG. 26 is a flowchart of steps in a method for teaching playing a musical instrument, in accordance with some embodiments of the disclosure.

FIG. 27 is a flowchart of steps in a method for selecting and configuring I/O devices to be used when teaching a user to play, in accordance with some embodiments of the disclosure.

FIG. 28 is a detailed flowchart of steps in a method for selecting and configuring I/O devices to be used when teaching a user to play, in accordance with some embodiments of the disclosure.

FIG. 29 is a block diagram of a system for selecting I/O devices when teaching a user to play, in accordance with some embodiments of the disclosure.

The principles and operation of an apparatus according to the present invention may be understood with reference to the figures and the accompanying description wherein similar components appearing in different figures are denoted by identical reference numerals. The drawings and descriptions are conceptual only. In actual practice, a single component can implement one or more functions. Alternatively or in addition, each function can be implemented by a plurality of components and devices. In the figures and descriptions, identical reference numerals indicate those components that are common to different embodiments or configurations. Identical numerical references (even in the case of using different suffix, such as 5 , 5 a , 5 b and 5 c ) refer to functions or actual devices that are either identical, substantially similar, or having similar functionality. It will be readily understood that the components of the present invention, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the apparatus, system, and method of the present invention, as represented in the figures herein, is not intended to limit the scope of the invention, as claimed, but is merely the representative embodiments of the invention. It is to be understood that the singular forms “a,” “an,” and “the” herein include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to “a component surface” includes reference to one or more of such surfaces. By the term “substantially” it is meant that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including for example, tolerances, measurement error, measurement accuracy limitations and other factors known to those of skill in the art, may occur in amounts that do not preclude the effect the characteristic was intended to provide.

Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms when used herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

Spatially relative terms, such as “inner”, “outer”, “beneath”, “below”, “right”, left”, “upper”, “lower”, “above”, “front”, “rear”, “left”, “right” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the example term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

In one example, it may be beneficial to adapt a music learning practice session to the specific user skill or ability. For example, a sheet music may be transcribed to reduce complexity of difficulty, either off-line according to estimated or expected user playing ability, or ‘on the fly’ in real-time during the practice session, to better adapt to the student skill level, such as by executing a flow chart 70 shown in FIG. 7 . A sequence of symbols that corresponds to playing of a musical piece on a musical instrument, such as in a sheet music form, is obtained as part of an “Obtain Sheet Music” step 71 . The sequence of symbols may be retrieved from a local memory, or from a storage of a remote server over the Internet. In case where the sequence of symbols (such as the sheet music) is received from a remote location as part of the “Obtain Sheet Music” step 71 , it may be locally stored for use as part of a “Store Version ‘5’” step 74 .

The obtained sequence, referred to as the ‘original’ sequence or version ‘5’, is analyzed for estimating the playing complexity or difficulty as part of an “Estimate Complexity” step 72 . The estimated (or calculated) complexity or difficulty level is stored as associated with the obtained sequence of symbols of the musical piece as part of a “Store Level” step 77 .

The estimation of the playing complexity or difficulty in the “Estimate Complexity” step 72 may be based on, may use, or may be according to, computationally quantifying performance difficulty as well as musical fidelity to the original score, and formulating the problem as optimization of musical fidelity under constraints on difficulty values using a statistical-modeling as presented in the article entitled: “ Statistical piano reduction controlling performance difficulty ” by Eita Nakamura and Kazuyoshi Yoshii [doi:10.1017/ATSIP.2018.18], published 2018 in SIP (2018), vol. 7, e13; analyzing the original music in order to determine the type of arrangement element performed by an instrument, then identifying each phrase and associated it with a weighted importance value as described in the article entitled: “ Automatic System for the Arrangement of Piano Reductions ” by Shih-Chuan Chiu, Man-Kwan Shan, and Jiun-Long Huang, published 2009 in the 11th IEEE International Symposium on Multimedia; analyzing the semantic content of music by focusing on symbolic music, i.e., sheet music or score, formulating difficulty level recognition as a regression problem to predict the difficulty level of piano sheet music, and recognizing the difficulty level of piano sheet music as described in the article entitled: “ A Study on Difficulty Level Recognition of Piano Sheet Music ” by Shih-Chuan Chiu and Min-Syan Chen, published December 2012 in the ISM '12: Proceedings of the 2012 IEEE International Symposium on Multimedia; Using Score Analyzer that is based on seven criteria to characterize technical instrumental difficulties in order to automatically extract the difficulty level of a MusicXML piece, and suggest advice based on a Musical Sign Base (MSB), as proposed in the article entitled: “SCORE ANALYZER: AUTOMATICALLY DETERMINING SCORES DIFFICULTY LEVEL FOR INSTRUMENTAL E-LEARNING” by Véronique Sébastien, Henri Ralambondrainy, Olivier Sébastien, and Noel Conruyt of IREMIA—Laboratoire d'Informatique et de Mathématiques, EA2525 University of Reunion Island, Saint-Denis, Reunion (FRANCE), published October 2012 in 13th International Society for Music Information Retrieval Conference (ISMIR 2012); using a systematic and objective approach to computational assessment of the complexity of a music score for any instrument, as introduced in the paper entitled: “ Musiplectics: Computational Assessment of the Complexity of Music Scores ” by Ethan Holder, Eli Tilevich, and Amy Gillick, published October 2015 in ONWARD '15 [ACM 978-1-4503-1995-9/13/10, http://dx.doi.org/10.1145/2508075.2514879], or determining the degree of difficulty by converting the score, which is expressed as a traditional music score, into electronic music sheet and calculating information about the elements needed to play sheet music by distance of notes, tempo, and quantifying the ease of interpretation, as proposed in the paper entitled: “ A Method for Measuring the Difficulty of Music Scores ” by Yang-Eui Song and Yong Kyu Lee [www.ksci.re.kr http://dx.doi.org/10.9708/jksci.2016.21.4.039] published April 2016 in the Journal of The Korea Society of Computer and Information Vol. 21 No. 4.

Alternatively or in addition, the estimation of the playing complexity or difficulty in the “Estimate Complexity” step 72 may be based on, may use, or may be according to, extracting features of the obtained sheet music as part of a “Extract Features” step 75 , and calculating difficulty characteristics on the mapped features as part of a “Calculate Difficulty” step 76 . The extracted features as part of the “Extract Features” step 75 may include notes or chords difficulty, pertaining to how the player fingers are required to be positioned, transition between notes/chords difficulty, defined Tempo, or associated decorations, such as excessive playing time or loudness.

In one example, the “Calculate Difficulty” step 76 may be based on, may use, or may be include, difficulty or skill characteristics, such as rhythmic difficulty, relating to how irregularly musical notes and chords are located in time, and what is the “speed” of music (the rate at which notes/chords should be played); motoric difficulty, relating to how difficult it is physically to produce the musical sounds using a musical instrument (usually mainly related to the complexity of the physical arrangement of user's fingers); harmonic difficulty, relating to how complex combinations of musical notes need to be played simultaneously (as chords) or sequentially (as very unpredictable melodies); or expressivity difficulty, relating to the user need to control and vary the loudness (“dynamics”) and timbre (“tone color”) of the musical sounds being produced. In one example, such analysis may be based on the U.S. Pat. No. 9,767,705 to Klapuri et al. entitled: “System for estimating user's skill in playing a music instrument and determining virtual exercises thereof”, which is incorporated in its entirety for all purposes as if fully set forth herein.

In one example, multiple levels of complexity or difficulty are defined, where Level ‘5’ corresponds to the most complex or difficult playing requirement, which is typically associated with the original arrangement that is obtained in the “Obtain Sheet Music” step 71 . In some cases, the complexity or difficulty estimated as part of the Estimate Complexity” step 72 may be appropriate for playing by an experienced player, but not suitable to be played by a beginner or student. In such a case, it may be beneficial to transcribe the sequence of musical symbols (or sheet music) to provide lower complexity or less difficult sequence of symbols, while retaining the general character or feeling of the original musical piece. In one example, four versions of sequences of musical symbols, referred to as versions ‘4’, ‘3’, ‘2’, and ‘1’, are generated, which are all transcribed or derived of the original version ‘5’. For example, version ‘4’ may be slightly simplified from the original version ‘5’, thus suited to more experienced players, ranging to version ‘1’, that is the most simplified version, which may be suitable to be played by novice players or beginners. While four versions are exampled, any number of transcribed versions may equally be produced, such as 1, 2, 3, 5, 7, 10, 15, 20, or more versions. The four versions are generated as part of a “Modify Arrangement” step 73 , and then may be stored for future use, such as storing version ‘4’ as part of a “Store Version ‘4’” step 74 a , storing version ‘3’ as part of a “Store Version ‘3’” step 74 b , storing version ‘2’ as part of a “Store Version ‘2’” step 74 c , and storing version ‘1’ as part of a “Store Version ‘1’” step 74 d.

The transcribing or modifying of the original version ‘5’ for producing simplified versions as part of “Modify Arrangement” step 73 , may be based on, may use, or may be according to, the article entitled: “ Statistical piano reduction controlling performance difficulty ” by Eita Nakamura and Kazuyoshi Yoshii [doi:10.1017/ATSIP.2018.18], published 2018 in SIP (2018), vol. 7, e13; the article entitled: “Automatic System for the Arrangement of Piano Reductions” by Shih-Chuan Chiu, Man-Kwan Shan, and Jiun-Long Huang, published 2009 in the 11th IEEE International Symposium on Multimedia; the U.S. Pat. No. 9,767,705 to Klapuri et al. entitled: “System for estimating user's skill in playing a music instrument and determining virtual exercises thereof”; or by available commercial application such as Ludwig (www.write-music.com/) and AnthemScore (www.lunaverus.com/).

An example of a system for online music learning by the person 36 of playing a musical instrument, such as the piano 83 , that is based on the client device 35 that communicates with a server device 23 a over the Internet 22 , is shown in an arrangement 80 shown in FIG. 8 , in an arrangement 80 a shown in FIG. 8 a , and in an arrangement 80 b shown in FIG. 8 b . The client device 35 may be identical to, similar to, part of, comprise, or integrated with, the computer system 10 shown in FIG. 1 or the client device 35 shown in FIG. 3 . In one example, the client device 35 , which may be a smartphone or a tablet computer, is used mainly for the purpose of interacting with the user 36 and the piano 83 , while the processing is mainly performed in the server device 23 a . The client device 35 communicates with the server 23 a over the Internet 22 using a wireless communication that involves the wireless transceiver 28 connected to the antenna 29 . As shown in the arrangement 80 a in FIG. 8 a , the teaching scheme is based on the server device 23 a sending visual data over a path 79 a (shown in FIG. 8 a ) that includes the Internet 22 , to be displayed to the user 36 via a display 81 in the client device 35 . The sound generated by the piano 83 is captured by a microphone 82 , where it is converted to a digital representation and sent to the server device 23 a over the Internet 22 over a path 79 b (shown in FIG. 8 a ), to be further processed therein.

The arrangement 80 shown in FIG. 8 , or the client device 35 , may be used with, integrated with, or used in combination with, a Virtual Reality (VR) system simulating a virtual environment to a person, and may be used by the VR system. The communication with the VR system may be wired or wireless, and the VR system may comprise a Head-Mounted Display (HMD).

The display 81 may be identical to, similar to, part of, comprise, or integrated with, the display 17 shown in FIG. 1 , and may be identical to, similar to, part of, comprise, or integrated with, the output component 34 of the arrangement 30 in FIG. 3 . The display 81 may consist of, or may comprise, a display screen for visually presenting information to the user 36 , and the display or the display screen may consist of, or may comprise, a monochrome, grayscale or color display, having an array of light emitters or light reflectors.

Alternatively or in addition, the display 81 or the display screen may consist of, or may comprise, an analog display having an analog input interface that supports NTSC, PAL or SECAM formats, and the analog input interface may comprise of RGB, VGA (Video Graphics Array), SVGA (Super Video Graphics Array), SCART or S-video interface. Furthermore, the display 81 or the display screen may consist of, or may comprise, a digital display having a digital input interface that is IEEE1394, FireWire™, USB, SDI (Serial Digital Interface), HDMI (High-Definition Multimedia Interface), DVI (Digital Visual Interface), UDI (Unified Display Interface), DisplayPort, Digital Component Video, or DVB (Digital Video Broadcast) interface.

For example, the display 81 or the display screen may be based on, or use, a Cathode-Ray Tube (CRT) display, a Field Emission Display (FED), an Electroluminescent Display (ELD), a Vacuum Fluorescent Display (VFD), an Organic Light-Emitting Diode (OLED) display, a passive-matrix (PMOLED) display, an active-matrix OLEDs (AMOLED) display, a Liquid Crystal Display (LCD) display, a Thin Film Transistor (TFT) display, an LED-backlit LCD display, or an Electronic Paper Display (EPD) display that is based on Gyricon technology, Electro-Wetting Display (EWD), or Electrofluidic display technology. Further, the display 81 or the display screen may consist of, or may comprise, a laser video display that is based on a Vertical-External-Cavity Surface-Emitting-Laser (VECSEL) or a Vertical-Cavity Surface-Emitting Laser (VCSEL). Furthermore, the display 81 or the display screen may consist of, or may comprise, a segment display based on a seven-segment display, a fourteen-segment display, a sixteen-segment display, or a dot matrix display, and may be operative to only display at least one of digits, alphanumeric characters, words, characters, arrows, symbols, ASCII, and non-ASCII characters.

The microphone 82 may be identical to, similar to, part of, comprise, or integrated with, the input component 38 of the arrangement 30 in FIG. 3 . The microphone 82 is configured to responds to audible or inaudible sound, and is an omnidirectional, unidirectional, or bidirectional microphone. The microphone 82 may be based on the sensing the incident sound-based motion of a diaphragm or a ribbon, or may consist of, or may comprise, a condenser, an electret, a dynamic, a ribbon, a carbon, an optical microphone, or a piezoelectric microphone. Further, the microphone 82 may consist of, may comprise, or may be based on, a microphone array for improving directivity.

The client device 35 may further comprise a sounder, such as a speaker 78 , that may be identical to, similar to, part of, comprise, or integrated with, the output component 34 of the arrangement 30 in FIG. 3 . The sound by any sounder such as the speaker 78 may be audible or inaudible (or both), and may be omnidirectional, unidirectional, bidirectional, or provide other directionality or polar patterns. The speaker 78 may be an electromagnetic loudspeaker, a piezoelectric speaker, an Electro-Static Loudspeaker (ESL), a ribbon or planar magnetic loudspeaker, or a bending wave loudspeaker. The sounder, such as the speaker 78 , may consist of, may comprise, may use, or may be based on, an electric sound source that may convert electrical energy into sound waves, and the electric sound source may be configured to emit an audible or inaudible sound using omnidirectional, unidirectional, or bidirectional pattern. Further, the sounder, such as the speaker 78 , may convert electrical energy to sound waves transmitted through the air, an elastic solid material, or a liquid, usually by means of a vibrating or moving ribbon or diaphragm.

Further, the sounder, such as the speaker 78 , may comprise, may use, or may be based on, an electromagnetic loudspeaker, a piezoelectric speaker, an electrostatic loudspeaker (ESL), a ribbon magnetic loudspeaker, a planar magnetic loudspeaker, or a bending wave loudspeaker. Furthermore, the sounder may consist of, may comprise, may use, or may be based on, an electromechanical scheme or a ceramic-based piezoelectric effect. In addition, the sounder may consist of, may comprise, may use, or may be based on, an ultrasonic transducer that may be a piezoelectric transducer, crystal-based transducer, a capacitive transducer, or a magnetostrictive transducer.

While piano 83 is exampled in FIG. 8 , any musical instrument may be equally used. For example, the musical instrument may consist of, or may comprise, a Soprano instrument such as a flute, a violin, a soprano saxophone, a trumpet, a clarinet, an oboe, or a piccolo. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, an Alto instrument, such as an alto saxophone, an French horn, an English horn, a viola, or an alto horn. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, a Tenor instrument, such as a trombone, a tenoroon, a tenor saxophone, a tenor violin, a guitar, or a tenor drum. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, a Baritone instrument, such as a bassoon, a baritone a saxophone, a bass clarinet, a cello, a baritone horn, or an euphonium. Alternatively or in addition, any musical instrument herein may consists of, or may comprise, a Bass instrument, such as a double bass, a bass guitar, a contrabassoon, a bass saxophone, a tuba, or a bass drum.

Further, the musical instrument may consist of, or may comprise, a string instrument that produces sound by means of vibrating strings, such as a guitar, an electric bass, a violin, a viola, a cello, a double bass, a banjo, a mandolin, an ukulele, or a harp. Any string instrument herein may consist of, or may comprise, a lute instrument in which the strings are supported by a neck and a bout, a harp instrument, in which the strings are contained within a frame, or a zither instrument, in which the strings are mounted on a body. Further, any string instrument herein be configured to be played by plucking, bowing, or striking. Furthermore, the string instrument any string instrument herein may be an acoustic instrument, or alternatively may comprise an electric amplification.

Furthermore, the musical instrument may consist of, or may comprise, a woodwind instrument that produces sound when the player blows air against a sharp edge or through a reed, causing the air within its resonator to vibrate, such as a flute, a piccolo, an oboe, an English horn, a clarinet, a bass clarinet, a bassoon, a contrabassoon, or a Saxophone. Any woodwind instrument herein may consist of, or may comprise, a flute that is configured to produce sound when air is blown across an edge, and any flute herein may consist of, or may comprise, an open flute, in which the player's lips form a stream of air which goes directly from the players lips to the edge, or a closed flute, in which in which the musical instrument has a channel to form and direct the air stream over an edge. Alternatively or in addition, any woodwind instrument herein may consist of, or may comprise, a reed instrument that is configured to produce sound by focusing air into a mouthpiece which then causes a reed, or reeds, to vibrate, and any reed instrument herein may consist of, or may comprise, a single reed instrument that uses a reed, which is a thin-cut piece of cane or plastic that is held against the aperture of a mouthpiece with a ligature, so that when air is forced between the reed and the mouthpiece, the reed vibrates to create a sound, a double reed instrument that uses two precisely cut small pieces of cane that are joined together at the base and are inserted into the top of the instrument to vibrate as air is forced between the two pieces, or a capped double reed instrument configured to vibrate when the player blows through a hole in a cap that covers the reed.

In addition, the musical instrument may consist of, or may comprise, a brass instrument that produces sound by sympathetic vibration of air in a tubular resonator in sympathy with the vibration of the player's lips, such as a Trumpet, a Cornet, a Horn, a Trombone, a Saxhorn, or a Tuba. Any brass instrument herein may consist of, or may comprise, a valved brass instrument that uses a set of valves operated by the player's fingers that introduce additional tubing, or crooks, into the instrument for changing its overall length, or a slide brass instrument that uses a slide to change the length of tubing.

Even more, the musical instrument may consist of, or may comprise, a percussion instrument that is configured to produce sound by being struck or scraped by a beater, such as a timpani, a snare drum, a bass drum, cymbals, a triangle, a tambourine, a glockenspiel, or a xylophone. Any percussion instrument herein may consist of, or may comprise, a pitched percussion instrument that produces notes with an identifiable pitch, or an unpitched percussion instrument that produces notes or sounds in an indefinite pitch.

Alternatively or in addition, the musical instrument may comprise, or may be based on, an acoustic, stringed musical instrument, configured so that the strings are struck by wooden hammers that are coated with a softer material, and the musical instrument may further comprise a keyboard that may consist of a row of keys, may be configured so that the performer presses down or strikes with the fingers and thumbs of both hands to cause the hammers to strike the strings. Any musical instrument herein may comprise, or may consist of, a piano, such as the piano 83 , which may comprise, or may consist of, a grand piano, an upright piano, or an electronic piano.

Further, any playing of any musical instrument herein may consists of, may comprises, or may be supplemented with, a vocal music, that may be with or without instrumental accompaniment, and may use lyrics, such as singing.

The display 81 in the client device 35 notifies the person 36 how to play the piano 83 by displaying a sequence 85 of musical symbols, which may be a simple training set for acquiring or enhancing the playing skills, or which may be according to musical notation standard or convention. For example, the sequence 85 shown on the display 81 may be part of, or an entire of, a musical piece. Further, the sequence 85 shown on the display 81 may be adapted to the specific played musical instrument, such as specially adapted to the piano 83 . In one example, the sequence 85 shown on the display 81 is a part of, or an entire of, a sheet music of a musical piece, and each of the musical symbols in the sequence may be according to a musical notation convention and typically includes lines, clefs, notes, and rests, and may notate pitch, tempo, meter, duration, or articulation of a note or a passage of music, and wherein the sequence.

The sequence may include one or more clefs that defines the pitch range or the tessitura of the symbol on which it is placed, one or more musical notes that denote a musical sound or the pitch and duration of a musical sound, one or more accidentals that denote a pitch or a pitch class, one or more key signatures that define the prevailing key of the music that follows, one or more time signatures that defines the meter of the music, one or more lines or a note relationships, one or more dynamics that indicate the relative intensity or volume of a musical line, one or more articulations that specify how to perform individual notes within a phrase or passage, one or more ornaments that modify the pitch pattern of an individual note, one or more octave signs, repetitions or codas, one or more pedal marks, or any combination thereof.

As part of the teaching session, the user 36 is expected to follow the displayed sequence 85 of the musical symbols, and to timely and accurately play by pressing the piano 83 keys (and pedals) in response to the shown symbols. In one example, a single symbol is shown at a time, such as the symbol 85 b shown in FIG. 8 a . Alternatively or addition, multiple symbols 85 a are displayed, such as the entire sheet music, or a part thereof of that includes one or more lines. In such a case, the symbol that is to be next played by the user 36 is specifically notified or marked on the display 81 , such as by changing color, enclosing it in a frame (such as a rectangular shape), or by pointing at it using an arrow 85 c , in order to clearly notify the player 36 what is the next symbol to be played. Preferably, the pace or tempo of displaying of the symbol in the sequence 85 are based on the skill level of the user 36 , on the tempo or pace associated with the musical piece that is represented by the sequence 85 (such as the tempo that is set or recommended by the creator or composer of the piece, or by a performer), or a combination thereof. For example, in case where the sequence 85 is associated with a musical piece that defines a specific tempo, the actual pace of displaying the symbols to the user 36 may be less than 95%, 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, or 10% of the defined tempo or pace, or may be at least 90%, 85%, 80%, 75%, 70%, 65%, 60%, 55%, 50%, 45%, 40%, 35%, 30%, 25%, 20%, 15%, 10%, or 5% of the defined tempo or pace.

The server device 23 a may comprise, may communicate with, or may be connected to, a memory 31 that may store one or more databases, such as for managing and handling a service of online music learning. The memory 31 may store a user's database 31 a that associates skill level values to various respective users, such as the user 36 , as shown in a table 86 (or in any other data structure) in FIG. 8 b . The skill level values may directly or indirectly map to the complexity or difficulty level that is calculated or estimated in the “Estimate Complexity” step 72 in the flow chart 70 . For example, a user value of ‘5’ corresponds to an expert that can play all musical pieces, a user value of ‘4’ corresponds to an less experienced user that is skilled enough to play all musical pieces associated with complexity or difficulty level of ‘4’ or less, and a user value of ‘1’ corresponds to a beginner or novice user that can properly play only musical pieces associated with complexity or difficulty level of ‘1’.

A top row 84 indicates that a left column 86 a includes identifiers of users, and a right column 86 b associates a skill level value to each of the users. In the example of the table 86 , a user identified as a User #1 84 a is associated with a skill level value of ‘1’, a user identified as an User #2 84 b is associated with a skill level value of ‘3’, a user identified as an User #3 84 c is associated with a skill level value of ‘2’, and a user identified as an User #4 84 d is associated with skill level value of ‘5’. While only exampled for four users, any number of users may be equally used.

The identifiers of the user shown in the column 86 a may typically comprise identifiers of the persons, such as the user 36 , who are using, or are expected to use, the system 80 . Such identifiers may comprise actual names, identification numbers (such as government identifying numbers), usernames, login names, e-mail address, telephone number, screennames (or screen names), account names, or nicknames, or any other identifiers of end users, which are the ultimate human users that operates, own, or use, the client device 35 . Alternatively or in addition, the user may choose to be anonymous, and will not provide any identifying details.

For example, a skill level value of ‘1’ may refer to a novice student, having no experience with music or with playing of the respective musical instrument, such as the piano 83 , and thus is expected to make many errors while playing. Such a user requires the slowest pace or tempo, and/or simplified music arrangement, to keep up with the sequence 85 shown on the display 81 . In contrast, a skill level value of ‘5’ may refer to an experienced or competent level student, having substantial experience with playing of the respective musical instrument, such as the piano 83 , and thus is expected to make no or minimal number of errors while playing. Such a user may easily track and