Aggregation and semantic modeling of tagged content

Conventional electronic communication systems generate a vast amount of short, unstructured text. This text may be present within electronic mail messages or Short Message Service (SMS) messages sent from a sender to one or more specific recipients, within microblog messages (e.g., Twitter “tweets”) posted to the World Wide Web, or within any other type of communication. Due to the lack of structure, and also due to the lack of content within each individual message, this text is not amenable to advanced storage, access and analysis techniques.

Hashtags are community-driven identifiers for specifying topics associated with microblog messages. Being both defined and selectively applied to a message by the message's author, hashtags are of limited use in effectively storing, accessing and analyzing microblog messages according to their semantic meanings.

FIG. 1 is a block diagram of a system according to some embodiments.

FIG. 2 is a flow diagram of a process according to some embodiments.

FIG. 3 is a block diagram of a system according to some embodiments.

FIG. 4 is a block diagram of an apparatus according to some embodiments.

The following description is provided to enable any person in the art to make and use the described embodiments and sets forth the best mode contemplated for carrying out some embodiments. Various modifications, however, will remain readily apparent to those in the art.

FIG. 1 is a block diagram of environment 100 . Environment 100 represents a logical architecture for describing some embodiments, and actual implementations may include more, fewer and/or different components arranged in any manner. The elements of environment 100 may represent software elements, hardware elements, or any combination thereof. For example, environment 100 may be implemented using any number of computing devices, and one or more processors within environment 100 may execute program code to cause corresponding computing devices to perform processes described herein.

Generally, each logical element described herein may be implemented by any number of devices coupled via any number of public and/or private networks. Two or more of such devices may be located remote from one another and may communicate with one another via any known manner of network(s) and/or via a dedicated connection.

Environment 100 includes message sources 110 , aggregator/modeler 120 and semantic models 130 . Message sources 110 are represented using a dashed line to indicate that any number and types of data sources may be used in conjunction with some embodiments. Non-exhaustive examples of message sources 110 include microblogging networks (e.g., Twitter), the World Wide Web, social networks (e.g., Facebook, LinkedIn), enterprise applications, groupware data (e.g., Microsoft Outlook files), one or more private intranets, one or more data warehouses, one or more text files, one or more spreadsheets and any other suitable message store and/or feed.

Aggregator/modeler 120 may operate to receive messages from message sources 110 . Aggregator/modeler 120 may therefore include appropriate interfaces required for communicating with each type of message sources 110 . As will be described below, aggregator/modeler 120 may generate semantic representations based on the text of the received messages. Briefly, aggregator/modeler 120 generates a semantic representation based on the text of one or more messages associated with a particular identifier (e.g., hashtag). Aggregator/modeler 120 may comprise an application running on an application platform which provides a suite of applications, such as but not limited to an enterprise resource platform. According to some embodiments, aggregator/modeler 120 is an application executed by a processor of a computing device.

Semantic representations 130 may comprise one or more data structures stored in a volatile or non-volatile memory device. As will be described below, each of semantic representations 130 is associated with a particular identifier (e.g., hashtag) according to some embodiments, and may advantageously represent a community-defined semantic meaning of the identifier. According to some embodiments, semantic representations 130 comprise semantic graphs, such as resource definition framework (RDF) graphs, which may be instantiated as an eXtensible Markup Language (XML) document according to known techniques. The semantic graphs may conform to Web Ontology Language or other suitable specifications that are or become known.

FIG. 2 comprises a flow diagram of process 200 according to some embodiments. Process 200 may be executed by aggregator/modeler 120 according to some embodiments. Process 200 and all other processes mentioned herein may be embodied in computer-executable program code read from one or more of non-transitory computer-readable media, such as a floppy disk, a CD-ROM, a DVD-ROM, a Flash drive, a fixed disk and a magnetic tape, and then stored in a compressed, uncompiled and/or encrypted format. In some embodiments, hard-wired circuitry may be used in place of, or in combination with, program code for implementation of processes according to some embodiments. Embodiments are therefore not limited to any specific combination of hardware and software.

FIG. 3 illustrates system 300 according to some embodiments. System 300 may comprise an implementation of environment 100 , but embodiments of environment 100 and of system 300 are not limited thereto. Examples of process 200 will be described below with respect to system 300 .

New text is initially received at S 205 . The new text and may be pushed from data sources 320 to aggregator/modeler 310 based on a subscription (e.g., a subscription to a particular data feed), requested by aggregator/modeler 310 from one or more data sources 320 , obtained by a crawler service of aggregator/modeler 310 , or otherwise received using any suitable mechanism that is or becomes known. According to some embodiments, aggregator/modeler 310 subscribes to (i.e., “follows”) all microblogs of data sources 320 , and the new text received at S 205 is a message posted to one of the followed microblogs.

Next, at S 210 , it is determined whether the received text includes an identifier. According to the present example, text parser 311 operates at S 210 to determine whether the received text includes a string beginning with “#” (e.g., #superbowl, #politics). Embodiments are not limited to this type of identifier. If the received text does not include an identifier, flow returns to S 205 to await next text.

Flow proceeds from S 210 to S 215 if the text includes a suitable identifier. At S 215 , it is determined whether the text includes a hyperlink. For example, text parser 311 may determine whether the text includes a Uniform Resource Locator pointing to a resource on the World Wide Web. It will be assumed for the moment that the text does not include a hyperlink, and therefore flow continues to S 220 .

At S 220 , it is determined whether a buffer associated with the identifier exists. In this regard, text buffers 312 may comprise memory areas (i.e., groups of memory locations) within a memory device, with each memory area being associated with a respective identifier. With respect to the identifiers mentioned above, text buffers 312 may comprise a first memory area associated with the identifier #superbowl and a second memory area associated with the identifier #politics.

If no buffer associated with the identifier of the new text exists, a new buffer is allocated and is associated with the identifier at S 225 . The text is then added to the new buffer at S 230 . If a buffer associated with the identifier was determined to exist at S 220 , the text is simply added to the buffer at S 230 . In this latter case, the associated buffer will include previously-received text which included the same identifier.

In a case that the text includes more than one identifier, the text is added to the buffers associated with each of the more than one identifier. Of course a situation may arise in which a buffer associated with one of the identifiers will already exist at S 220 and a buffer associated with another one of the identifiers will have to be allocated at S 225 .

At S 235 , it is determined whether a threshold associated with the buffer has been reached. The threshold may comprise a memory size, a number of stored texts, and/or any other characteristic based on which a buffer may be evaluated. If the threshold has not been reached, flow returns to S 205 . Accordingly, flow cycles as described above to allocate buffers associated with identifiers and to store received text associated with identifiers in the appropriate buffers.

If it is determined at S 235 that a buffer has reached the buffer threshold, a semantic representation of all the text in the buffer is generated at S 240 . Generally, and according to some embodiments, the semantic representation indicates two or more entity types and one or more instances of each of the entity types that exist within the text. Therefore, generation of the semantic representation may consist of identifying entities which are present in the text, determining an entity type of each entity, and creating a semantic digest which lists the determined entity types and the entities of each type which exist in the text.

Prior to generation of the semantic representation at S 240 , the text may be transformed into a standardized format (e.g., Atom) which is suitable for input to the component which generates the semantic representation. According to the FIG. 3 embodiment, this component is semantic classifier 313 . For example, semantic classifier 313 may receive text of the buffer from text buffers 312 and implement OpenCalais in order to generate an XML document describing an RDF graph therefrom.

It will now be assumed that the text received at S 205 included a hyperlink. Flow therefore proceeds from S 215 to S 245 to retrieve text associated with the hyperlink. According to one example, the hyperlink points to a Web page, and text parser 311 operates to request the Web page from data sources 320 . As is known, the hyperlink could point to any network-accessible file of any format. Flow then proceeds to S 240 to generate a semantic representation based on the text received at S 205 and on any text retrieved at S 245 using the hyperlink. The generated semantic representation is associated with the identifier which was included in the text received at S 205 .

According to some embodiments, the semantic representation is generated after S 245 based on the text received at S 205 , any text retrieved at S 245 using the hyperlink, and the contents of any message buffer associated with the identifier. In some embodiments, the text retrieved at S 245 is simply added to the associated message buffer, and flow continues therefrom to S 235 and proceeds as described above.

After generation of a semantic representation at S 240 , it is determined whether the identifier associated with the semantic representation is already associated with a previously-stored semantic representation at S 250 . In this regard, graph store 315 stores semantic representations generated by process 200 , each of which is therefore associated with an identifier (e.g., #superbowl). Aggregator agent 314 receives a semantic representation generated by semantic analyzer 313 and determines whether an identifier associated with the semantic representation is associated with a semantic representation stored within graph store 315 .

If not, the semantic representation is stored in association with the identifier at S 255 . If the identifier is associated with an existing semantic representation stored within graph store 315 , the current semantic representation is added to the existing semantic representation at S 260 . According to some embodiments, S 260 includes applying a UNION operation to the current semantic representation and the existing stored semantic representation to create a third semantic representation which is stored in graph store 315 in association with the identifier. Other methods for creating a new semantic representation based on two semantic representations may be employed at S 260 .

Flow returns to S 210 from S 250 /S 255 to receive new text and continue as described above. The semantic representations of graph store 315 may be utilized for any number of purposes, including but not limited to discovering content related to particular identifiers, trend analysis, and data-mining or indexing of microblogs.

FIG. 4 is a block diagram of apparatus 400 according to some embodiments. Apparatus 400 may comprise a general-purpose computing apparatus and may execute program code to perform any of the functions described herein. Apparatus 400 may comprise an implementation of elements of aggregator/modeler 110 and/or 310 . Apparatus 400 may include other unshown elements according to some embodiments.

Apparatus 400 includes processor 410 operatively coupled to communication device 420 , data storage device 430 , one or more input devices 440 , one or more output devices 450 and memory 460 . Communication device 420 may facilitate communication with external devices, such as data sources including Web servers of the World Wide Web. Input device(s) 440 may comprise, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, knob or a switch, an infra-red (IR) port, a docking station, and/or a touch screen. Input device(s) 440 may be used, for example, to enter information into apparatus 400 . Output device(s) 450 may comprise, for example, a display (e.g., a display screen) a speaker, and/or a printer.

Data storage device 430 may comprise any appropriate non-transitory storage device, including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives and flash memory), optical storage devices, Read Only Memory (ROM) devices, etc., while memory 460 may comprise Random Access Memory (RAM).

Aggregator/modeler 432 of data storage device 430 may comprise program code executable by processor 410 to provide functions described herein, including but not limited to process 200 . Embodiments are not limited to execution of these functions by a single apparatus. Message buffers 434 include memory areas associated with particular identifiers, and may also or alternatively be implemented in memory 460 . Graph store 436 may associate semantic representations associated with particular identifiers as described above.

Data storage device 430 may also store data and other program code for providing additional functionality and/or which are necessary for operation thereof, such as device drivers, operating system files, etc.

Embodiments described herein are solely for the purpose of illustration. Those in the art will recognize other embodiments may be practiced with modifications and alterations to that described above.