I needed to capture the mcu ADC sensor output to a file.

This is just the Nerdkit's Tempsensor code with a IR distance sensor.

bretm wanted to do some statistical analysis on the sensor readings. Very interesting.

Anyway, it took me a couple of hours to be able to capture the serial signal and to make a file with the data:

#!/usr/bin/python
import serial

s = serial.Serial("/dev/cu.PL2303-0000101D", 115200)

f = open("TempSensor.txt", "w")

while True:

    x = s.readline() 
    print "",x                     #output (print) to monitor
    f.write(x)                     #capture serial output/input
    f.flush()                      #write to file

f.close()

I would appreciate any comments especially about my method. None of the forum threads I found on the Internet offered a similar solution. None of them used the flush so I do not know if that is a good idea or what.

I had to work my way through a number of errors starting out with the "no indented block" error, who knew indentation blocks mattered.

So what do you think.

Should I add this to the Library it seems a lot of people have needed Python serial capture.

Ralph

Python uses indenting just like C uses curly braces, so the indentation is critical.

I would be surprised if the flush is needed. It may actually slow down the I/O.

Well I broke the code in trying to clean it up!!

It didn't work until I added the flush.

The program executed but nothing was written to the file.

Now I have to figure out what I did, hey I actually have automated backup dam I bet I can recover my working code from last night.

This will be the first time I ever actually used a backup to recover a file. I am using the Mac Time Machine.

Ralph

It is not the code it was my serial connection.

So now what should I do instead of the flush?

The serial data was displayed on the monitor (print "",x) but the text file was not written.

Also how should the print statement be formatted?

Ralph

Oh, the f.close never happens which is why there isn't any output without f.flush. It's an infinite loop, so I assume you're killing the app manually or interrupting it in some way. So the output buffer never gets flushed unless you explicitly flush. I guess that's what you need to do if you're intending to terminate abnormally.

I think you can also open the file without buffering by specifying a buffer size of 0 as a third argument to f.open.

Can you just do print(x) instead of print "",x?

Thanks bretm, I have not figured out how to terminiate "normally" so I just close the running monitor application.

Since this type of program does not need a GUI it would just be running continuously in the background in a infinite loop I am not sure how one would terminate it normally. Most of the Python programs I have found use a GUI with a Stop button.

If I hit f.close I would have to repeatedly reopen so that seems to waste cycles.

How would I get output without f.flush?

Like I said I put this together as my first Python program so I have barely looked at the documentation, thanks for the feed back.

Ralph

Instead of using an infinite loop, only loop for the number of bytes you want to capture. Your last dataset was around 50,000 so just execute the loop that many times. Then it will fall out of the loop and close the file. Will python do something like

int i=0;
while(i++<50000)

If you don't want to use f.flush you should be able to open the file with buffering disabled. You can open the file without buffering by specifying a buffer size of 0 as a third argument to f.open.

Ok here is my latest first attempt:

#!/usr/bin/python
import serial
import sys

s = serial.Serial("/dev/cu.PL2303-0000101D", 115200)

f = open("Sample1.txt", "w",0)

while True:

    x = s.readline() 
    try:
        print(x) 
        f.write(x)

    except (KeyboardInterrupt, SystemExit):
        break

f.close()

I have the disabled buffering ( f = open("Sample1.txt", "w",0)).

I got rid of the f.flush().

And the thread is broken with Ctrl-C

except (KeyboardInterrupt, SystemExit):
        break

Essentially the "program" (Python Interpreter) is still running but the thread is broke.

Where do I go from here I would like to either restart the thread or kill the interpreter.

Boy the Python documentation is really good but seems to lack depth.

Ralph

Hi Ralph,

Ctrl+D will kill the interactive shell. The way you have structured your program though, Ctrl+C should trigger the KeyboardInterrupt, get caught by your except caluse, and then break out of the while loop which will then exit the program. It will only do this if you happen to hit ctrl+C while it was trying to do the print or the write though. If you were waiting in the s.readline() then the exception will not be caught. If you really want control of the ctrl+c action you want to do something like

#!/usr/bin/python
import serial
import sys

s = serial.Serial("/dev/cu.PL2303-0000101D", 115200)

f = open("Sample1.txt", "w",0)
try:
  while True:
     x = s.readline() 
     print(x) 
     f.write(x)

except (KeyboardInterrupt, SystemExit):
  #do something
  print "exception caught"
  f.close()

Notice how the control flow stays entirely within the try: block, then when ctrl+c is hit it generates the exception and I can catch it.

From an overarching system point of view though, I still prefer the original code you came up with

#!/usr/bin/python
import serial

s = serial.Serial("/dev/cu.PL2303-0000101D", 115200)

f = open("TempSensor.txt", "w")

while True:

    x = s.readline() 
    print "",x                     #output (print) to monitor
    f.write(x)                     #capture serial output/input
    f.flush()                      #write to file

f.close()

This code will spend most of its time waiting for input from the mcu on the s.readline(), so having the f.flush() at the bottom actually seems like a good idea to me. This lets python buffer your write, then write it out to the file at the end of each loop when you flush. The only slight problem you do run into is if you ctrl+c to break the loop you may have been in the middle of a write (which is actually an issue in the other code too), its just something you have to be aware of when opening the file for viewing or processing. The code is also just more concise.

Humberto

Thanks everyone, now how would I do a live file write?

Right now the file is not written to until I exit the while loop.

I do not know how long I could read the serial data into memory before I had problems.

I was hoping the flush would happen every cycle of the while, but apparently not.

I could use a timer and break the while every so often but then I would need to restart the program.

A live feed would be much better.

Should I be using an append instead of a write?

Ralph

Hi Ralph,

On my laptop just running f.flush() is enough to force the write to the file, and I've never had a problem with that in the past. You might have have to use os.fsync() like explained here http://docs.python.org/library/stdtypes.html#file.flush, something like

#!/usr/bin/python
import serial,os
 
s = serial.Serial("/dev/cu.PL2303-0000101D", 115200)
 
f = open("TempSensor.txt", "w")
 
while True:
 
    x = s.readline() 
    print "",x                     
    f.write(x+"\n")                
    f.flush()                      
    os.fsync(f)

f.close()

Why don't you just use 'C'?

BM

BM. using C is on my list. I just thought since Python is referenced so often here on the Nerdkits forums for doing any PC interface and it is what the Nerdkits guys use in their tutorials I ought to at least learn how to do a serial interface (data logging).

Speaking of Python data logging take a look at this tutorial now that makes my head spin. But it really looks good just no explanation on how to use it :-( for a knownothing).

There is a logging method built into newer version of Python that one can use:

import logging, logging.handlers

But I am still looking for a simple explanation on how to use it.

Of course I do not know how to do it in C so your help is welcome. I think "Data Logging" should be added to the new Library. If you post some simple C code for serial data logging I'll post it to the Library also.

I will pursue Humberto's suggested os.fsync(f) that looks interesting.

Ralph

Now this works very nicely:

#!/usr/bin/python
import serial,os

s = serial.Serial("/dev/cu.PL2303-0000101D", 115200)

f = open("TempSensor.txt", "w")

while True:
        x = s.readline() 
        #print(x)                         #Remove the comment to see output in monotor program
        f.write(x)
        f.flush()                      
        os.fsync(f)

f.close()

Relatively simple, at least I can understand it.

This type program is meant to just be running in the background.

Of course that leads to more questions.

Is the Python interpreter multi-threaded?

If I was running this program from the command line could I run other Python programs also?

Or would I need to combine (include?) the other programs into one consolidated program. This of course would defeat using the continuos while loop. Life sure isn't simple.

You have to kill the app manually as bretm noted. Using the KeyboardInterrupt does work but does not seem necessary to me.

Hey Bobba I Googled "C serial" and "C serial logging". There certainly does not appear to be a 12 line method of doing Serial capture in C or at least I did not find one. Everything I did find was OS dependant so there was no cross platform C method also.

I sure would appreciate a simple C program for serial capture.

Ralph

Hi Ralph,

There are ways to run multi threaded programs in Python, but in this application you don't need it. Everytime you invoke the Python it spawns a new environment in its own process. Other python programs can be spawned from other command line shells and will be just another process on your system.

One caution I would have when writing a program that is meant to "run in the background" is that an infinite while loop like you have could take over the CPU and keep running wildly over and over. In this case you should spend a good amount of time in the s.readline() just waiting for input. s.readline() should be implemented correctly such that you are not actually consuming the CPU during that time, but its something to be aware of.

Humberto

Ralph,

I just wondered why you were using Python. When I get a routine done and tested in 'C', I'll post it.

BM

Thanks Boba looking forward to it.

I think every language I have looked at has some sort of serial interface it would be interesting to put together a collection of different language's implementations.

From what I have seen JAVA is the only language that that does not have native serial support.

I have worked with some Visual Basic and C# and even a little F# maybe I coudld replicate what I have done in Python.

Ralph