Method for generating web code for UI based on a generative adversarial network and a convolutional neural network

This application is a national stage application filed under 35 U.S.C. 371 based on International Patent Application No. PCT/CN2020/085880, filed on Apr. 21, 2020, which claims priority to Chinese Patent Application No. 201910562411.2, filed with the CNIPA on Jun. 26, 2019, disclosures of which are incorporated herein by reference in their entireties.

The present disclosure belongs to the field of software development and, in particular, relates to a method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN).

The explosive growth of the scale and complexity of software products makes the rapid development of software more challenging. Particularly in the early stage of software development, the designer designs a prototype diagram and implements the prototype diagram with codes, which is a very heavy workload and is inefficient. Through the research on the automatic generation of software codes, the developer can accelerate the development process, implement software functions more quickly, and finally deliver their software products quickly. It can be seen that the research on the automatic generation of software codes becomes increasingly important.

At present, there are many technologies for automatic code generation, including template-based Freemarker, Extensible Stylesheet Language Transformations (XSLT) and velocity, model-driven model-driven architecture (MDA) and model-driven development (MDD), object-relational-mapping-based object-relational mapping (ORM) and model-view-controller (MVC), document annotation-based Annotation and XDoclet, and agent-dynamic-class-based aspect-oriented programming (AOP), PROXY and ASM. These automatic code generation methods only play a partial auxiliary role in the software development process and cannot replace a certain part of the software development, and thus have a rather limited effect on accelerating the software development. In addition, these automatic code generation methods require learning in related fields first, and then they can be mastered and applied in actual development. As the relatively-high use threshold, these methods are not friendly to most developers for use, so that they have poor universality and cannot be applied to actual development on a large scale.

The present disclosure aims to provide a low-cost, high-accuracy and wide-application-range method for generating web codes for a user interface (UI).

The technical solution to achieve the object of the present disclosure is a method for generating web codes for a UI based on a generative adversarial network (GAN) and a convolutional neural network (CNN). The method includes steps described below.

In step 1, a mapping relationship between display effects of a HyperText Markup Language (HTML) element and source codes of the HTML element is constructed.

In step 2, object recognition is performed on an image I of the UI to recognize a location of an HTML element in the image I.

In step 3, complete HTML codes of the image I are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2.

In step 4, a similarity Sim 1 between manually-written HTML codes and the generated complete HTML codes is obtained for the image I of the UI.

In step 5, a similarity Sim 2 between the image I of the UI and an image I 1 generated by the generated complete HTML codes is obtained.

In step 6, minimax game is performed on generative networks G in the generative adversarial network (GAN) and discriminative networks D in the GAN to obtain an equilibrium solution from all possible generative networks G and discriminative networks D, so that the generative networks G generate HTML codes having a same distribution as the manually-written HTML codes, where the generative networks G in the GAN are formed by step 2 and step 3, and the discriminative networks D in the GAN are formed by step 4 and step 5; the Sim 1 in step 4 and the Sim 2 in step 5 are balanced as a Sim 0 by the equilibrium solution; whether the Sim 0 is greater than a set threshold t is determined; and in a case where the Sim 0 is less than t, steps 2 to 5 are repeated; in a case where the Sim 0 is not less than t, step 7 is executed.

In step 7, an image-to-HTML-code generation model M is obtained according to a finally obtained generative G and discriminative network D of the GAN after the training in step 6, and inputting a to-be-processed UI image is input into the model M so as to obtain corresponding HTML codes.

The present disclosure is described below in detail in conjunction with the drawings.

FIG. 1 is a flowchart of a method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN) according to the present disclosure;

FIG. 2 is an input web design diagram demo1.png according to an embodiment of the present disclosure;

FIG. 3 is a Button object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

FIG. 4 is a Divider object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

FIG. 5 is a Flag object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

FIG. 6 is an Icon object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

FIG. 7 is a Breadcrumb object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

FIG. 8 is a Message object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

FIG. 9 is a Checkbox object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure; and

FIG. 10 is a screenshot after demo2.html is run according to an embodiment of the present disclosure.

Referring to FIG. 1 , the method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN) of the present disclosure includes steps described below.

In step 1, a mapping relationship between display effects of a HyperText Markup Language (HTML) element and source codes of the HTML element is constructed.

In step 2, object recognition is performed on an image I of the UI to recognize a location of an HTML element in the image I.

In step 3, complete HTML codes of the image I are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2.

In step 4, a similarity Sim 1 between manually-written HTML codes and the generated complete HTML codes is obtained for the image I of the UI.

In step 5, a similarity Sime between the image I of the UI and an image I 1 generated by the generated complete HTML codes is obtained.

In step 6, minimax game is performed on generative networks G in the generative adversarial network (GAN) and discriminative networks D in the GAN to obtain an equilibrium solution from all possible generative networks G and discriminative networks D, so that the generative networks G generate HTML codes having a same distribution as the manually-written HTML codes, where the generative networks G in the GAN are formed by step 2 and step 3, and the discriminative networks D in the GAN are formed by step 4 and step 5; the Sim 1 in step 4 and the Sime in step 5 are balanced as a Sim 0 by the equilibrium solution; and whether the Sim 0 is greater than a set threshold t is determined; in a case where the Sim 0 is less than t, steps 2 to 5 are repeated; in a case where the Sim 0 is not less than t, step 7 is executed.

In step 7, an image-to-HTML-code generation model M is obtained according to a finally obtained generative G and discriminative network D of the GAN after the training in step 6, and a to-be-processed UI image is input into the model M so as to obtain corresponding HTML codes.

Compared with the related art, the present embodiment has significant advantages. 1) Computer vision technologies are fully used for preforming object recognition processing on a UI image in advance, so as to improve a mapping relationship between display effects of an HTML element and source codes of the HTML element and make the mapping relationship more reliable and accurate. 2) The mapping relationship between display effects of the HTML element and the codes is acquired, and the mapping relationship may be continuously extended, so that the effect of the model of the present disclosure is finally improved, the accuracy of the codes generated for the image is improved, and the application range becomes wider. 3) An image-to-HTML-code conversion model is acquired. When the conversion model is used in practice, as long as a corresponding image is input, the final code results can be obtained. Therefore, some parts in the actual development can be replaced, the actual use cost becomes lower, and the application field becomes wider.

In an embodiment, step 1 specifically includes the step described below. Feature maps of HTML elements are extracted by using the convolutional neural network (CNN), and a one-to-one correspondence is established between the feature maps of the HTML element and HTML codes. Common HTML elements are: Elements (Button, Container, Divider, Flag, Header, Icon, Image, Input, Label, List, Loader, Placeholder, Rail, Reveal, Segment and Step), Collections (Breadcrumb, Form, Grid, Menu, Message and Table), Views (Advertisement, Card, Comment, Feed, Item and Statistic) and Modules (Accordion, Checkbox, Dimmer, Dropdown, Embed, Modal, Popup, Progress, Rating, Search, Shape, Sidebar, Sticky, Tab and Transition).

In an embodiment, step 2 in which the object recognition is performed on the image I of the UI to recognize the location of the HTML element in the image I specifically includes steps described below.

In step 2-1, a feature map of the image I of the UI is extracted by using the convolutional neural network (CNN). Step 2-1 specifically includes the step described below. The feature map is extracted by using a convolution layer, a pooling layer and a Rectified Linear Unit (ReLU) layer, and edge extension is performed during multiple times of convolution to ensure that a size of an input matrix and a size of an output matrix do not change after the multiple times of convolution.

In step 2-2, an object proposal region is generated by using a region proposal network (RPN).

In step 2-3, a proposal feature map corresponding to the object proposal region is extracted by a region of interest (ROI) pooling layer of the pooling layer.

In step 2-4, a final accurate location of the object proposal region, that is, the location of the HTML element in the image I of the UI, is detected by a classification layer and based on the proposal feature map corresponding to the object proposal region.

In an embodiment, step 2-2 in which the object proposal region is generated by using the region proposal network (RPN) specifically includes steps described below.

In step 2-2-1,

ceil ⁡ ( a ⁢ width ⁢ of ⁢ an ⁢ original ⁢ image SW ) × ceil ⁡ ( a ⁢ height ⁢ of ⁢ the ⁢ original ⁢ image SH ) × Z

proposal anchors are set in the original image I, where SW represents a scale of the width of the original image, SH represents a scale of the height of the original image, and Z is an integer.

In step 2-2-2, which anchors are foreground points having an object and which anchors are background points having no object are determined by using the CNN.

In step 2-2-3, bounding box regression processing is performed on the foreground points having the object so as to obtain an object proposal region corresponding to the foreground points having the object. Exemplarily, SW=16, SH=16, and Z=9.

In an embodiment, step 2-3 in which the proposal feature map corresponding to the object proposal region is extracted by the ROI pooling layer in the pooling layer specifically includes steps described below.

It is assumed that a size of the object proposal region is M×N, and a scale of the object proposal region is SW×SH.

In step 2-3-1, a dimension of the object proposal region is mapped to

In step 2-3-2, a feature map region corresponding to the dimension-mapped object proposal region is horizontally divided into grid cells each of which has a size of pooled w ×pooled h .

In step 2-3-3, max pooling processing is performed on a feature map in each of the grid cells so as to obtain the proposal feature map corresponding to the object proposal region.

In an embodiment, step 2-4 in which the final accurate location of the object proposal region, that is, the location of the HTML element in the image I of the UI, is detected by the classification layer and based on the proposal feature map corresponding to the object proposal region includes steps described below.

In step 2-4-1, for each of the proposal feature map, which HTML element a respective object proposal region belongs to is calculated through a fully connected layer and a softmax classifier to obtain a rough location of the respective object proposal region, and a probability vector of the rough location is output.

In step 2-4-2, based on the probability vector, a location offset of the respective object proposal region is obtained by using the bounding box regression method to regress to a more accurate object region.

Further, step 3 in which the complete HTML codes of the image I are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2 specifically includes steps described below.

In step 3-1, a feature map corresponding to an object proposal region whose accurate location is obtained in step 2 is extracted by using the CNN.

In step 3-2, based on the mapping relationship in step 1, HTML codes corresponding to the feature map in step 3-1 are matched according to a value of the feature map in step 3-1 so as to obtain HTML codes corresponding to all of HTML elements in the image I.

In step 3-3, all HTML code blocks are combined to generate the final complete HTML codes.

Further, step 4 in which the similarity Sim 1 between the manually-written HTML codes and the generated complete HTML codes is calculated for the image I of the UI specifically includes steps described below.

In step 4-1, a set S 1 of attribute class values in the manually-written HTML codes and a set S 2 of attribute class values in the generated complete HTML codes are respectively extracted.

In step 4-2, the similarity Sim 1 is obtained by using a Jaccard coefficient through the following formula:

S 1 represents the set of the attribute class values in the manually-written HTML codes, and S 2 represents the set of the attribute class values in the generated HTML codes.

Exemplarily, the image I 1 is generated by the generated complete HTML codes in step 5 includes two manners described below.

The generated complete HTML codes are run by using a modern standard browser, and a screenshot of display results is taken to obtain the I 1 .

Alternatively, the I 1 is obtained by rendering the generated complete HTML codes by using an imgkit package of Python.

Exemplarily, step 5 in which the similarity Sim 2 between the image I of the UI and the image I 1 generated by the generated complete HTML codes is obtained in step 5 specifically includes the step described below. The similarity Sim 2 is obtained based on a histogram, image template matching and perceptual hashing algorithm of image similarity algorithms.

Embodiment

The method for generating web codes for a UI based on a generative adversarial network (GAN) and a convolutional neural network (CNN) of the present disclosure includes contents described below.

In step 1, a mapping relationship between display effects of an HTML element and source codes of the HTML element is constructed. Step 1 specifically includes the step described below.

Feature maps of HTML elements are extracted by using the CNN, and a one-to-one correspondence is established between the feature maps of the HTML element and HTML codes.

Common HTML elements include Elements (Button, Container, Divider, Flag, Header, Icon, Image, Input, Label, List, Loader, Placeholder, Rail, Reveal, Segment and Step), Collections (Breadcrumb, Form, Grid, Menu, Message and Table), views (Advertisement, Card, Comment, Feed, Item and Statistic) and Modules (Accordion, Checkbox, Dimmer, Dropdown, Embed, Modal, Popup, Progress, Rating, Search, Shape, Sidebar, Sticky, Tab and Transition).

In the embodiment, an input web design diagram demo1.png is shown in FIG. 2 . In the embodiment, the one-to-one mapping relationship of Button, Divider, Flag, Icon, Breadcrumb, Message and Checkbox is defined as shown in Table 1 below.

In step 2, object recognition is performed on an image 1 to recognize a location of an HTML element in the image 1 . Step 2 specifically includes steps described below.

In step 2-1, a feature map of the image 1 of the UI is extracted by using the CNN. Step 2-1 specifically includes the step described below. The feature map is extracted by using a convolution layer, a pooling layer and a Rectified Linear Unit (ReLU) layer, and edge extension is performed during multiple times of convolution to ensure that a size of an input matrix and a size of an output matrix do not change after the multiple times of convolution.

In step 2-2, an object proposal region is generated by using a region proposal network (RPN).

In step 2-2-1,

ceil ⁡ ( a ⁢ width ⁢ of ⁢ an ⁢ original ⁢ image SW ) × ceil ⁡ ( a ⁢ height ⁢ of ⁢ the ⁢ original ⁢ image SH ) × 9

proposal anchors are set in the image 1 .

In step 2-2-2, which anchors are foreground points having an object and which anchors are background points having no object are determined by using the CNN.

In step 2-2-3, bounding box regression processing is performed on the foreground points having the object so as to obtain an object proposal region corresponding to the foreground points having the object.

In step 2-3, a proposal feature map corresponding to the object proposal region is extracted by an ROI pooling layer in the pooling layer.

It is assumed that a size of the object proposal region is M×N.

In step 2-3-1, a dimension of the object proposal region is mapped to

In step 2-3-2, a feature map region corresponding to the dimension-mapped object proposal region is horizontally divided into grid cells each of which has a size of pooled w ×pooled h .

In step 2-3-3, max pooling processing is performed on a feature map in each of the grid cells so as to obtain the proposal feature map corresponding to the object proposal region.

In step 2-4, a final accurate location of the object proposal region is detected by a classification layer, that is, which HTML element a respective object proposal region belongs to is calculated through a fully connected layer and a softmax classifier by using the obtained feature map, and a probability vector cls_prob is output. At the same time, a location offset bbox_pred of the respective object proposal region is obtained by using the bounding box regression method again for regression of a more accurate object region.

In the embodiment, the finally obtained object images are shown in Table 2 below.

In step 3, complete HTML codes of the image 1 are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2.

According to the object images in Table 2 above, codes are searched for in the mapping relationship in Table 1, and the codes are obtained as shown in Table 3 below.

The above codes are stitched to obtain following complete codes demo2.html:

<body>

<button class=“ui button”>Follow</button>

<div class=“ui divider”></div>

<div>

<i class=“ae flag”></i>

<i class=“france flag”></i>

<i class=“myanmar flag”></i>

</div>

<div>

<i class=“question circle icon”></i>

</div>

<div class=“ui breadcrumb”>

<a class=“section”>Home</a>

<i class=“right angle icon divider”></i>

<a class=“section”>Store</a>

<div class=“active section”>T-Shirt</div>

</div>

<div>

<div class=“ui compact message”>

<p>

Get all the best inventions in your e-mail every day. Sign up

now!

</p>

</div>

</div>

<div>

<div class=“ui checkbox”>

<input type=“checkbox” name=“example” />

<label>Make my profile visible</label>

</div>

</div>

</body>.

In step 4, a similarity Sim 1 between manually-written HTML codes demo1.html and the generated complete HTML codes demo2.html is obtained for the image 1 . Step 4 specifically includes steps described below.

In step 4-1, a set S 1 of attribute class values in the manually-written HTML codes and a set S 2 of attribute class values in the generated complete HTML codes are respectively extracted.

In step 4-2, the similarity Sim 1 is obtained by using a Jaccard coefficient through the following formula:

S 1 represents the set of the attribute class values in the manually-written HTML codes, and S 2 represents the set of the attribute class values in the generated HTML codes.

In the embodiment, the manually-written HTML codes demo1.html are:

<!DOCTYPE html>

<html lang=“en”>

<head>

<meta charset=“UTF-8” />

<meta name=“viewport” content=“width=device-width, initial-scale=1.0” />

<meta http-equiv=“X-UA-Compatible” content=“ie=edge” />

<title>Document</title>

<link

rel=“stylesheet”

href=“https://cdn.jsdelivr.net/npm/semantic-ui @2.4.2/dist/semantic.min.css”

/>

<script

src=“https://cdn.jsdelivr.net/npm/semantic-ui @2.4.2/dist/semantic.min.js”></script>

</head>

<body>

<button class=“ui button”>Follow</button>

<div class=“ui divider”></div>

<div>

<i class=“ae flag”></i>

<i class=“france flag”></i>

<i class=“myanmar flag”></i>

</div>

<div>

<i class=“question circle icon”></i>

</div>

<div class=“ui breadcrumb”>

<a class=“section”>Home</a>

<i class=“right angle icon divider”></i>

<a class=“section”>Store</a>

<i class=“right angle icon divider”></i>

<div class=“active section”>T-Shirt</div>

</div>

<div>

<div class=“ui compact message”>

<p>

Get all the best inventions in your e-mail every day. Sign up now!

</p>

</div>

</div>

<div>

<div class=“ui checkbox”>

<input type=“checkbox” name=“example” />

<label>Make my profile visible</label>

</div>

</div>

</body>

</html>.

From the above, it can be obtained that:

S1={“ui button”, “ui divider”, “ae flag”, “france flag”, “myanmar flag”, “question circle icon”, “ui breadcrumb”, “section”, “right angle icon divider”, “section”, “right angle icon divider”, “active section”, “ui compact message”, “ui checkbox”, };

S 2 ={“ui button”, “ui divider”, “ae flag”, “france flag”, “myanmar flag”, “question circle icon”, “ui breadcrumb”, “section”, “right angle icon divider”, “section”, “right angle icon divider”, “active section”, “ui compact message”, “ui checkbox”, }.

In this way, it can be obtained that the similarity Sim 1 satisfies that

In step 5, a similarity Sim 2 between the image 1 of the UI and an image generated by the generated complete HTML codes is obtained. Step 5 specifically includes steps described below.

The method for generating the image by the generated complete HTML codes is described below. The generated complete HTML codes are run by using a modern standard browser, and a screenshot of display results is taken; or, the generated complete HTML codes are rendered by using an imgkit package of Python.

The similarity Sim 2 is obtained based on a histogram, image template matching and perceptual hashing algorithm of image similarity algorithms. In the embodiment, the codes demo1.html are run, and the display effect of the codes are saved as shown in FIG. 10 , and the Sim 2 between the image 1 and FIG. 10 is obtained as 100%.

In step 6, minimax game is performed on generative networks G in the generative adversarial network (GAN) and discriminative networks D in the GAN to obtain an equilibrium solution from all possible generative networks G and generative networks D, so that the generative networks G generate HTML codes having a same distribution as the manually-written HTML codes, where the generative networks G in the GAN are formed by step 2 and step 3, and the discriminative networks D in the GAN are formed by step 4 and step 5; the Sim 1 and the Sim 2 are balanced as a Sim 0 by the equilibrium solution; whether the Sim 0 is greater than a set threshold t is determined; and in a case where the Sim 0 is less than t, steps 2 to 5 are repeated; in a case where the Sim 0 is not less than t, step 7 is executed.

In the embodiment, t is set to 95%; and Sim 1 =100% and Sim 2 =100%, so that the balanced Sim 0 is obtained as 100%. At this time, Sim 0 =100% and t=95%, so that Sim 0 is greater than t, and the next step is executed.

In step 7, an image-to-HTML-code generation model M is obtained according to a finally obtained generative G and discriminative network D of the GAN after the training in step 6, and a to-be-processed UI image is input into the model M so as to obtain corresponding HTML codes.

According to the method of the present disclosure, an image-to-HTML-code generation model M can be obtained. A to-be-processed UI image is input into the model, and then corresponding HTML codes can be generated. Therefore, the model has stronger universality and generality and can replace some parts in the actual development, and thus the actual use cost is lower and the application field is wider.