Aerocet531S serial read

2014-09-10

####Serial read from Aerocet 531S###

Aerocet 531S(A531S) has serial communication and commanding interface through USB connection. By connecting USB to Linux ubunut 12.04, running the program serial port terminal and giving the command such as S starts the A531S. These commands are elaborated in its user manual.
There number of other commands to view the data and get more about the A531S operation as per its user manual.
To make the process of running the machine and entering the command in Python to start the machine or read output is the aim of this note.
A simple google search resulted in this and the program was tested with A531S, this is the edited code considering the options for A531S. This program was returning nothing from A531S, but editing the line no. 30 into ser.write(input + '\r') return some charter by sending the provided command in its user manual.
So made a searching for the source code behind the program serial port terminal. It ended in the example page of pyserial a library for python based serial communication, the program named as miniterm.py
The program miniterm.py was downloaded and it was run as per its instruction by entering the command python miniterm.py -p '/dev/ttyUSB0' -b 9600 --parity=N, it gives welcome message and by entering ENTER and typing the command in the line starts with * controls the A531S such as starting smaple etc.
Based on this, new search was made to Scriptify this program and interface of A531S command entering to start sample etc. Based on this it is found that the python module subprocess is good way to start another python script and made this script to run that and it could make upto the point of starting the miniterm.py program not enabling the Command entering interface.
Then made a search to know sequential command entering by python and found that subprocess module PIPE is requiered for that explained here1 ,here2. Based on this a script was made with different iterative command in it to run this but ends with error saying termios.error: (25, 'Inappropriate ioctl for device') and it goes to several other errors of input/ouput kind.
Then made a search to resolve above issues and learn that, serial communcation is very vast subject and requiees the idea of how the machine is programmed to accept the serial command, this siwhy the simple serial command ios failing with A531S. While doing it in Arduino the arduion side also programmed to accpet the command. Any way the working of miniterm is indicative that the conversin can be understood to scriptfy the control of A531S. More info i avilable form here
Later found a simplified version for miniterm.py, as this file. This script can be run by this command python miniterm2.py -p '/dev/ttyUSB0' -b 9600, controlling the device works fine. This script is easily understandable, the lines 17 to 30 is addressing windows implementation and lines 32 to 49 addresses Linux implementation. In Linux implementation, the function getkey play major role in generating the command interface and line 68 to 82 function writer communicate the commands with the device for control, such as by entering S getkey converts to some form and function writer communicate it with device to start the sampling. the lines 152 to 163 execute these functions.
So it seems, executing this program in subprocess could attain the objective. But it gives error as of last complex miniterm. But is is learnt that the lines from 34-41 necessitate the command entering in the program. it is also worth noting this from here “Standard output and standard error (commonly abbreviated stdout and stderr) are pipes that are built into every UNIX system. When you print something, it goes to the stdout pipe; when your program crashes and prints out debugging information (like a traceback in Python), it goes to the stderr pipe. Both of these pipes are ordinarily just connected to the terminal window where you are working, so when a program prints, you see the output, and when a program crashes, you see the debugging information. (If you’re working on a system with a window-based Python IDE, stdout and stderr default to your “Interactive Window”.) “
It is found that subprocess is not suitable for the process of interactive shell script, the miniterm is a interactive shell script. This inference is based on this for non suitablity of subprocess and this for intro to pexepct.
Using this code import pexpect command = "python miniterm3.py -p '/dev/ttyUSB0' -b 9600" child=pexpect.spawn(command) child.expect('\r\n') print child.before child.sendline ('S') it is able to run the A531S sampling routine automatically, it takes iteratuive(some time it works some time not) so try has to made in code, or other wise A531S has to refreshed with empty lines in mintierm by child.sendline (''). Have to do for logging the output from the sample

to iterate the empty line in pexepct, from

        for _ in itertools.repeat(None, 3):
                child.sendline ('')

To change the sample mode from count to mass or wise versa, for mass mode child.sendline ('MM 1'), for count mode child.sendline ('MM 0').
The pexepct is a tool which exactly simulate the terminal behavior of the tool, in our case the program myterm.py. So knowing the behvious of myterm.py is requiered to scriptfy that by pexpect. the program myterm.py needs atleast two RETURNS for get into * line, where it accepts the command. Typing S and RETURN(press enter) starts the machine by giving response start sample. The A531S samples for 1 minutes and write the values in new * line and it stays there arbitaraly. Pressing another two to three RETURN only get into new line of * and then only the program accpts next commands. Here the empty line of pexpect has to be iterate for three times. For to get the sample results it has to be mentioned about the line * by child.expect('*') but this ends in error of
```
        raise error, v # invalid expression
        sre_constants.error: nothing to repeat
```
So tried child.expect (['%',pexpect.EOF]) this ends in unresponsive terminal line.
So made a wild attempt of doing print child.before it gets the sample result in this state 22:40:43,01237800,00103470,00023920,00000340,00000060,+029,052,001,060,000,*04203 with out any date files contained in the output. So it has to be adjusted programmatic.

The normal routine of comamnds are

        import pexpect
import itertools
        command = "python miniterm3.py -p '/dev/ttyUSB0' -b 9600"
        child=pexpect.spawn(command)
        child.expect('\r\n')
        for _ in itertools.repeat(None, 3):
                child.sendline ('')
        child.sendline ('MM 0')
        child.sendline ('S')
ctO = child.before
for _ in itertools.repeat(None, 3):
                child.sendline ('')
time.sleep(30)
child.sendline ('MM 1')
        child.sendline ('S')
masO = child.before

It is found that the command ctO = child.before and masO=chile.before is not working and after a large iteration, it is understood that the child.before only showing just oneline above the terminal. So based on the number of line executed in the terminal inculding iterative loops command, it has to count and looped to execute this command to reach the most updated sample output line. It was devised by using the below command to loop through commands and make cursor to reach the most updated sample output.
```
    for _ in itertools.repeat(None, 5):
         child.expect('\r\n')
         print child.before
```

Based on this, program is rewritten as follows

        import pexpect
import itertools
        command = "python miniterm3.py -p '/dev/ttyUSB0' -b 9600"
        child=pexpect.spawn(command)
#For Count mode
        for _ in itertools.repeat(None, 5):
                child.sendline ('')
        child.sendline ('MM 0')
        child.sendline ('S')
        for _ in itertools.repeat(None, 8):
         child.expect('\r\n')
         print child.before
child.expect('\r\n')
ctO=child.before
        #For Mass mode
        for _ in itertools.repeat(None, 5):
                child.sendline ('')
        child.sendline ('MM 1')
        child.sendline ('S')
        for _ in itertools.repeat(None, 9):
         child.expect('\r\n')
         print child.before
        child.expect('\r\n')
masO=child.before
        child.close()

To make the above script more useful especially by having colocation sampling with Dylos air quality monitor it is better to make the proragm as def(). It made to def() as below

        def a531s():
                        command = "python miniterm3.py -p '/dev/ttyUSB2' -b 9600"
                        child=pexpect.spawn(command)
                #For Count mode
                        for _ in itertools.repeat(None, 5):
                            child.sendline ('')
                                child.sendline ('MM 0')
            -----------------------------
            -----------------------------
        a531s()

Initially had error of port can’t be open in Raspberry pi. RPi was having already connected two Dylos air quality mointor. The ls /dev has to be checked for port and, if it is given /dev/ttyUSB2 it has to be trailed for its correctness. With this the script run and the sample was taken for both count and mass mode. But faced problem in getting the data read by serial into print or save it in csv out of def a531S. Based on this for naming global variable and its non pythonic nature, this for pythonic way of getting variable out of def. Based on that a simple python test code was written and it is working,

             def areo():
                     x='abc'
                     y='cba'
                     reads = {'a':x,'b':y}
                     return reads
            df=areo()
          print df['a']

So the script was modified as per the above test, but failed to get any values in the variables assigned. It was become a prolonged problem, various methods such as child.logfile as by here didn’t given any way out. As far as the script, the function running and taking sample in different mode one after one and saving data into the A351S internal storage is working but the sample output can’t be accessed if it is in script. Surprisingly the data can be written out in executing the command one by one but never happen in complete script method, that is saving whole command in a file and executing it.
Based on the experience of getting data out of a531S, for time being, attempt for real time monitoring with it is stopped for time being. The data has to be collected manually by entering the command in miniuterm.py. There is an attempt also made to edit the miniterm.py to be bare metal script to control and receive data it also didn’t turn working the script is here.
With the minimal functionality of the written script in controlling a531s is used for collocated sampling of dylos and a531s, the script was modified to run simultaneous using threading based on this and the script used is this and crontab was setup to run this script every 10 minutes.
Trying to run the script sampler.py in crontab -e, it was unresponsive, so made a search about this issue and find this and tryed to execute the command using /home/pi/aerocet/sampler.py ends with this error -bash: /home/pi/aerocet/sampler.py: /usr/bin/python^M: bad interpreter: No such file or directory, this error was searched and found this and asks to execute this command perl -pi~ -e's/\r$//' sampler.py, this solved the last error and now it is working with the attempted code, the serial read from dylos is erroneous.
Other issue faced was non working of crontab. Tried this note and it seems the cron daemon is working but no log is written. Made a shell script as like below and tried instead of directly running the python script
```
        #!/bin/sh
        cd /home/pi/aerocet
        sudo python sampler.py
```

finally set the date of the Rpi, suddenly crontab started to work. It was setup in sudo crontab -e with this command */10 * * * * /home/pi/aerocet/run.sh &>/home/pi/aerocet/cron.log 1. Another issue faced was erroneous serial read from dylos monitor. Using miniter.py it was mostly erroneous, using the written script, for first 20 samples (data reading every 120 seconds) it was reading first serial read data(<0.5 um paticles) only. This was getting improved when I connected the dylos serial USB directly tot the RPI instead of routing it through USB hub.

####usefull links####

to install pexpect, http://www.pythonforbeginners.com/systems-programming/how-to-use-the-pexpect-module-in-python
to delay a python script, http://stackoverflow.com/questions/510348/how-can-i-make-a-time-delay-in-python
to save the terminal output into file and sys.stdout in console in simulatneously, http://www.dtcenter.org/HurrWRF/users/tutorial/2014_tutorial/tutorialpractical2014/compile_exercise.htm, sample ./compile nmm_real 2>&1 | tee compile_nmm.log
notes on pexpect, http://stackoverflow.com/questions/15316914/reading-output-from-pexpect-sendline http://stackoverflow.com/questions/13590578/print-and-pexpect-logging http://stackoverflow.com/questions/9370886/pexpect-if-else-statement http://stackoverflow.com/questions/15211011/python-pexpect-returning-the-command-and-the-output-from-the-command http://pexpect.readthedocs.org/en/latest/api/pexpect.html#spawn-class http://stackoverflow.com/questions/19433046/python-pexpect-spawn-object-flush http://stackoverflow.com/questions/19143360/python-writing-to-and-reading-from-serial-port http://stackoverflow.com/questions/419163/what-does-if-name-main-do http://pexpect.readthedocs.org/en/latest/commonissues.html http://pexpect.sourceforge.net/doc/
mulitple deivce serial into single computer note, http://stackoverflow.com/questions/16949369/serial-communication-one-to-one
notes on stdout, http://stackoverflow.com/questions/1218933/can-i-redirect-the-stdout-in-python-into-some-sort-of-string-buffer http://stackoverflow.com/questions/2513479/redirect-prints-to-log-file
more insight into sys.stdout.flush(), http://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method