A test script generator StimuliSelector

Supposed, a script is necessary to control the test. This script may be:

For a simple explanation a simple test file with parameters and a XML file is presented here:

Adequates in XML:

The content of this files are an example, it is user specific.

You need a file which contains exactly this texts but with place holder for the values. The place holder are written in the form: <&path.variable>. But additionally a frame for the text is necessary. So the template for the text file for this example is (file testfile_text.jzTc):

The area between <::> and the closing <.> is the whole original text with the placeholder. The text is organized in JZtxtcmd as 'subtext' syntax, with calling arguments, see ../../JZtxtcmd/html/JZtxtcmd.html#Topic.JZtxtcmd.script.sub.. The template script can also contain comments written as ##line end comment which are not part of the generated result. A simple # is not a comment because often the # is need to generate comment lines. Of courses that requirement or feature is regarded in JZtxtcmd.

It is also possible in XML (file testfile_xml.jzTc):

JZtxtcmd is proper also and especially for generation of results which uses elaborately writing with < …​ > for example XML, but also some other script languages e.g. the Mathworks-specific "tlc" scripts ("target language compiler" control script). The distinction to XML style is: The placeholder start with <& which is never used in XML as necessary direct text. Some control constructs start with <: also not used in XML. So that sequences can be used immediately in the template. But nevertheless this character sequences can be written as <:<&> etc. to produce it as output, see ../../JZtxtcmd/html/JZtxtcmd.html#Topic:.JZtxtcmd.text.transcript..

From where come the values?

You should define tables, (see file stimuliTables.jzTc in the appended example):

The List is type of ../../Java/docuSrcJava_vishiaBase/org/vishia/cmd/JZtxtcmdExecuter.ListMap.html. It can also be accessed as Map with a key. The variable which’s content builds the key is designated after the @ in the list’s head.

Any line of this tables is a java.util.Map container with the name of the variables as key. The line should contain a variable named descr for the GUI tool. Furthermore any variables are admissible holding values. But each line of a table should contain the same named variables.

The tables build groups of selectable test cases, which can be used for manual selection, and also for test generation.

The GUI with this given tables looks like:

The tables in the table script are shown and are selectable. One can select a specific test case with this tables and press [gen selection] to get the generated files for this case.

It is possible to assemble some test cases and press [gen test cases] to generate all files for all tests, or to control execution of test.

For now the tool creates only the textual files. If you want to use the tool independent of your test system, you can use the generated files in the test system.

But it is also possible to interact with the test system. This is done with the capabilities of the JZtxtcmd script language, or also with batch files. You can generate a script which invokes executables by itself or sends and receives messages for example via socket communication. This depends only on your design of the scripts.

The evaluation of test results is not a functionality of the Stimuli Selector. But the scripts can help for evaluation.

It means, before establish automatic test cases, reasonable combinations should be found. Manual tests with expertise of the results can help to find the combinations.

You should think about what is to test, think about which combinations are proper, and get experience with some combinations. It is gathering experience with manual handling.

Hence you can select one combination with the tables, press the button [gen selection], maybe improve the test files, study the behavior of your system in test, and also complete the tables and the characteristics of the tables.

The idea is: Assembling some test cases on demand and study manually the behavior. Then gather the test cases in a expression:

If you have a test selected in the tables and you click [add sel] button then you get a select expression in the text box right of [gen test cases]:

This expression contains table numbers and associated keys of the lines of test cases.

In the example secondly the shown case v3 was selected in the table, and then pressed [add sel] again. Then this case is also added. If you select other lines in the other tables and click [add sel] this lines are added too. You can check some test results manually pressing [gen selection], then [add sel]. But:

This expression means, this two cases from table 1 are combined with the two cases from table 2 and the other selections in the tables. It describes 2*2*3*2*3 = 72 test cases, the combination of all. If the tables are longer, it is sometimes too much. Hence there are more possibilities:

That are two independent combinations, separated with the : colon. It is 2*2 + 2*2 = only 8 test cases. You may have the expectation that you want to test the v1 combination only with the given ones, for example only with C3 and the other cases from table 1 especially with C1 only. - to reduce the amount of combination for a fast overview test.

You can entry this expression manually, but also write the colon, :, set the cursor after it, select in table and press [add sel]. The expression builder detects that you are in a separated part of the expression. Because it is empty, it takes all lines of the table.

The test case select expression allows skillful combinations, some selections with some others. See the next expression:

The first line dedicates 4 cases in combination v2, v3, B1, B3 and a combination v1, B2. The + means a 'add' of a case.

The & means 'and with', a selection in other tables to build a combination. Here the 5 combination above with table 1 and 4 are combined with the second line, they are two combinations with t1 and t2 and A1 and a third one with t2 and A3, In the manual or considered tests this combinations may be proper.

Last not least this is combined & 'and with' the C1 of the 5_th table.

The expression can be written in one line or more, white space is the principle

In result the following tests are combined:

To fill such more complex expression you have the following possibilities:

The syntax for the select expression is:

It is written as ZBNF expression. It is similar BNF or EBNF. {…​? .. } means a repetion with separator after the ?. <#?table> is a number with the meaning (semantic) "table". <$?selItem> is an identifier (letters, digits and underlyer) with the given semantic meaning. […​] is optional, hence the semicolon is optional. But it should be written. The ZBNF syntax has the advantage that semantic information are given in the formal syntax string ("Zemantic BNF").

Look on the example:

uses the + also here. It means 3 cases: t1_A1, T2_A1 and T2_A3.

The example contains four variants of using the selection. You have:

Change the called routine to …​_A as shown then the following is called:

This routine generates for all test cases a set of files with different names. The name for the generated test file(s) is build from the table content. Calling the same routine as in [gen selection] the files are generated.

This routine can be seen as principle pattern:

In the example you see the access to the line of the table values for the test case using the key testcase[0].sel, adequate lineTexts = texts.get(testcase[1].sel) for the other line of table.

Alternatively you can generate also a file as organization frame for all tests with content controlled by the test cases.

Sometimes it is sufficient to have one file which contains the execution of all test cases. This is either with all data of test cases, or alternatively with separated files for the test case values. The following example shows both.

The sub routine btnGenTestcases_B in StimuliSelector.jzT.cmd generates one file for all test cases. The file template itself is testall_text.jzTc and also testall_xml.jzTc. It is called with the opened file:

Because it is called as subtext inside the text expression for the file <+fText>…​<.+> it writes to the file. But the testall_text is a normal sub routine, not a sub text:

The text in <::>…​<.> is outputted inside the text expression, that is writing to the file. But the sub routine contains also control statements, to organize the execution for all test cases. Similar it is for XML, see file testall_xml.jzTc in the example.

Of course you can generate one file for all test cases and extra files as control file for each test case, in combination with the content of sub btnGenTestcases_A and sub btnGenTestcases_B.

The next example comes from a Simulink usage, not from this example. See https://www.vishia.org/smlk/html/SmlkTimeSignals/SmlkTimeSignals.html.

Result of this generation is a Matlab script file. You see the content with place holder after the == on start of lines. The generated Matlab script file organizes while running (if the test is started) the creation of a Matlab array. This array contains only a String which is used as arguments for test script generation (line ===testcase{ix, 1}=…​). For example it is generated as Matlab script:

This is done for all test cases as result of preparing with TestConditionCombi.prepareTestCases(select, 6);

After that action the generated Matlab script contains a for loop for all elements of the Matlab array. The for loop is executed in the generated Matlab script. It does the following:

The createStimuli.jzT.cmd generates the current used test script via JZtxtcmd (Java call) as command called inside the Matlab script. This file contains:

The first lines presents a batch file (cmd file). Java is called. It ends on exit /B.

The next lines after ==JZtxtcmd== contains a main() routine which is invoked firstly. This main routine gets the keys which are comes from the Matlab array as 'user arguments' written as -u:key generated via -u:<&var.sel> in the script above.

It is a nesting of some scripts, which allows using the capability of test organization in Matlab (with an array for arguments and possibility of a system(cmd) call, and then start the simulation via sim(model).

The details how the genStimuli(…​) works are not presented here, see link above. It is a normal file generation using the template with placeholder approach.

Socket communication is an elegant approach to data exchange. A simple string message can be sent and received with a UDP telegram. The queue to store messages is already given with the 'Telegram Stack' on driver level.

However, if the queue is to work properly, the socket communication should be initialized once at startup (open socket) and remain open. Otherwise, the partner can send a message and only after that the communication will be opened. Then the message is lost. This can occur if the test system does not have the socket communication approach natively, but should execute it via a command line call. If the socket cannot remain open for the entire time, it should be timed.

The socketCmd.exe given in the example can be used if the test system has no native socket support but can call system commands. The problem for remain opening is solved in the following way:

The Stimuli Selector GUI transmits only a command after receiving, and the socketcmd.exe transmits firstly, then wait for receiving. It means the socket is open and waiting for receive in the moment if the Stimuli Selector GUI transmits. That is proper.

If the test system is started firstly while the socket on the Stimuli Selector GUI is not opened, because the [gen test cases] is inactive, then the transmition of a cmd by socketcmd.exe is ignored, but the receiving after them recognizes the missing destination from the transmisson and returns error 10054. This is a standard behavior for socket communication. In this situation the socketcmd.exe returns with errorlevel=254 which can be evaluated for 'wait for GUI'.

The socketcmd.exe is programmed in C/++ with emC sources (available via Github) inside the project IDE/IDE/VS15_emCapplications/emCapplications.sln. It offers:

On GUI side it is lesser complicated because Java can deal natively with sockets. The socket is only open if the thread for [gen test cases] does run. It may be also a decision to open the thread for the whole running of the GUI.

For the Gui the class ../../Java/docuSrcJava_vishiaRun//org/vishia/communication/SocketCmd_InterProcessComm.html offers the socket communication opportunities:

Using in the StimuliSelect_Example

The socketCmd.exe is stored in libs/ but gotten from its vishia-repository.

The file StimuliSelector.jzT.cmd contains

You can change the called routine, change M against A or F to use the other approches. Here M is described.

The button routine quests an internal variable soRx. It is null if no communication is pending, then genTestCaseThread(…​) is called.

If it is pressed again during test cases are generating respectively the thread is running and waiting for interaction, the socket communication is closed. Following the receive routine for communication is aborted and the thread is finished. This is important, because for interaction it is not clarified what the partner does. Anytime the control over the whole process should be given.

In this routine a thread is created in JZtxtcmd: ../../JZtxtcmd/html/JZtxtcmd.html#Topic:.JZtxtcmd.thread. which calls getTestCastThread_M(…​).

The tread routine is used also for immediately invocations for nightly tests, see chapter Automatic execution of tests:

In this first part the communication via socket is set up with the shown socket communication class. The address string can be better defined at the start of the script to be able to change it if needed. The port number is manually determined in coordination with the interaction partner. Here a local communication ("127.0.0.1" is the 'loop back' address) is used, instead any network card with known IP-V4 can be addressed too.

In expecting of a proper communication the generation of test files is started via calling prepareTestCases(…​), see chapter above. The files for the first test cases are generated:

In the loop soRx waits for a messae from the test system. But before an error handling is done.

The quest soRx.hasError() is especially for the first invocation to detect an open error for the communication. Then the thread is aborted with a message, cannot work. This is common when perhaps an incorrect network has been selected, the port is in use, etc. The last one is especially given if the tool is started twice!

If no error is given, then the Stimuli Selector waits for a request from the test system via jztc.envar.soRx.waitRx();.

The test system may running in a loop (waiting for the Stimuli Selector) or not. It is emulated by the following batch file TestSystem_Msgs.bat:

If this batch was started without running [gen test cases], it loops via errorlevel 254. To prevent a high CPU load because of spinning the socketcmd.exe waits here with 2000 ms timeout. It means if an error occurs it waits 2 seconds (with operation system thread capabilities) before return.

The thread in the Stimuli Selector GUI is continued in case of receiving step with:

It changes the color and text of the buttons to see in the GUI what’s happen. But this is only done if the variable stimuliSelector is available in jztc.envar. This is because the same routine is also used for Automatic execution of tests without the GUI.

Then the files for the testcase are generated calling exactly the same routine as in the button [gen selection].

After a seconds-sleep for this example only to show the green color of the button, but may be in praxis for timing conditions of the test system, the message test is sent to the test system. After them the color of the button is changed again, and it continues in the loop which starts with waiting to step from the test system, see above.

If test is received from the test system, socketcmd.exe returns with errorlevel 1 (the first -rx token) and continues on:

For this example the test is only emulated by type of the content of one generated file and a wait. Because MS-Windows has no delay capabilities the socketcmd.exe is here used only to delay, an intrinsic proper capability.

A serial communication is similar. But a serial driver is not yet contained in the Stimuli Selector GUI. A simple way to do this is: Program in C/++, run it all the time, and exchange data via sockets with this special program and the Stimuli Selector GUI.

This is another possibility and socket and query files should combined sometimes.

In a test system it is often possible to query the existence of files and to remain in a loop if they are not present. This seems to be the best approach for the test files itself. But there is a pitfall. When the test file is started to be written, it already exists. But it is not ready for use. The test system should wait until the file is really ready. But this may not be queried in a unique way. Hence: Using the test files itself is a lesser good decision.

Files can be used as semaphore. If there are existing, it is a unique state. They can be deleted, then they are not existing, even correct.

Files can be addressed in a network. It is possible to realize a network communication with these semaphore files.

Another important idea is: Files should not be stupidly created or deleted. They should be renamed. The advantage is: Renaming only changes the directory entry in the file system. There is no necessity to allocate new space on the media.

But there is also a pitfall: Sometimes, if a process was aborted, the files are messed up. To prevent exactly this, the following operation is written in the vishiaJava_Base.jar component:

FileFunctions.renameCreate(File dir, String src, String dst, boolean bException)

With this functionality the principle of file-semaphores are used in the Stimuli Selection GUI (JZtxtcmd) in the following way:

It cleans a situation from pre-usage. The existence of any semaphore file may be possible. Expected is: genScripts/idle.msg exists from pre-usage or by first usage it is not existing. After execution, this is present and no other genScripts/.msg file exists. Note: You can use also the ‘’ as part of line, for example msg*_FromXtoY.txt. All files with the wildcard will be cleaned up.

A similar line. Because the situation is cleaned, the idle.msg is renamed to test.msg It is better to use this invocation instead a simple rename command, because - the situation may be unknwon again. Any other tool or person can touch on the file system.

The other side, the test system: Have a look on a test.do script used on a VHDL test system (https://www.aldec.com/en/products/fpga_simulation/active-hdl):

This script part waits for the file test.msg but also for finish.msg. It runs in a "goto"-loop elsewhere, it is a "spinning". It test.msg is detected, it can be usual rename without failure in the currently running situation. Writing !ren is necessary here for this script language, ! to start a system command.

But in continuing this script the back message works with an event via socket:

In this loop in the script also an evaluation of test results can be done.

Behavior in the StimuliSelect_Exampl

The file StimuliSelector.jzT.cmd contains

You can change the called routine, change M against A or F to use the other approches. Here F is described. That routine starts with:

It is similar to btnGenTestcases_M(…​) as in chapter Possibility to interact via socket but soRx is here a simple Boolean variable which is removed.

The genTestCaseThread(…​) creates a thread in JZtxtcmd: ../../JZtxtcmd/html/JZtxtcmd.html#Topic:.JZtxtcmd.thread

The tread routine is used also for immediately invocations for nightly tests, see chapter Automatic execution of tests:

The start of the routine cleans up a possible mess situation with the file semaphores. It deletes all *.msg files and renames only one to idle.msg.

In expecting of a running test thread the generation of test files is started via calling prepareTestCases(…​), see chapter above. The files for the first test cases are generated:

In the loop soRx waits for a messae from the test system. But before an error handling is done.

The test files are created for the first test case, independent of the running test system. After them the idle.msg is renamed to test.msg. But if the situation may not clarified in any case, instead rename also the renameCreate(…​) routine is used. It is a java routine with less additional effort for this additional features.

The test system may be started before or after them. It waits in any case of the file test.msg:

The socketcmd.exe is here used only for a thread-wait behavior. It goes in a loop till the file is existent.

If the test.msg is recognized, it executes the test:

The test is here also emulated by a wait (via socketcmd.exe after type. Ater the test the test.msg is renamed to step.msg. This is rcognized by the Stimuli Selector GUI:

Here in JZtxtcmd also the existence of a file is checked, with Java capabiities. The File:"name" creates a java.io.File istance, which*s exists() operaiton is invoked. Additional the jztc.envar,soRx is quest. It is a boolean variable with true, which is set to null on abort, see above. The following sleep(…​) prevents too high CPU load on spinning.

The generation loop is continues if soRx is still true.

After finish the loop in the Stimuli Selector GUI the file is renamed to finish.msg which is checked by the test system, see above.

This can be included either in the test system or in the GUI. How to evaluate test results - it depends deeply on the test system. Often results are stored in files, the content of the files should be evaluated, and over all results an average or overview should be built, and a report of faulties. This can be done of course immediately in the test system, but also integrated in the Stimuli Selector GUI. The benefit of the Stimuli Selector GUI is: It is programmed in Java (from JZtxtcmd Java routines can be called simple). And - some things can be done - Adaptions - immediately in JZtxtcmd without necessity of Java compiling.

The example does not have such capabilities, look at concrete applications.