I would like to be able to read and write the raw data from an SD card using just a computer (no microprocessor) forgetting the FAT32. I would like to do this in basic C. I can look at the raw data with a program called winhex. Long run I want someone who is less computer savvy to run a C script that will store certain parameters on SD card (where I say so) that the microprocessor would use. Then store data on it allowing me to never have to connect directly to a computer. I’m having a hard time getting started with this program partly because I’m not even sure what to search for (I keep getting sights trying to sell me stuff). So if anyone has knowledge of the CreateFile() and ReadFile() you help would be great. Here is my code so far with problem area commented (Windows XP, CodeBolcks10.05) mostly from some people at cprogramming.com

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <math.h>
#include <Windows.h.>
#include <conio.h>

#define BUFFER_SIZE 8

int main(void)
{
    HANDLE hFile;
    DWORD dwBytesRead = 1;
    char ReadBuffer[BUFFER_SIZE+1] = {0};

    hFile = CreateFile("\\\\.\\E:", GENERIC_READ|GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);

    if (hFile == INVALID_HANDLE_VALUE)
    {
        printf("Could not open file (error %d)\n", GetLastError());
        return 1;
    }

    /// Code isent getting past next step or reading data ///

    // Read one character less than the buffer size to save room for
    // the terminating NULL character.
    if( FALSE == ReadFile(hFile, ReadBuffer, BUFFER_SIZE-2, &dwBytesRead, NULL) )
    {
        printf("Could not read from file (error %d)\n", GetLastError());
        CloseHandle(hFile);
        return 2;
    }

    if (dwBytesRead > 0)
    {
        ReadBuffer[dwBytesRead+1]='\0'; // NULL character

        printf(TEXT("Text read from %s (%d bytes): \n"), dwBytesRead);
        printf("%s\n", ReadBuffer);
    }
    else
    {
        printf(TEXT("No data read from file %s\n"));
    }

    CloseHandle(hFile);

    return 0;
}

So it’s (I'm) having a hard time reading data. I like trouble shooting programming problems by breaking code up but I don’t even know where to start with this problem, any ideas? Thanks

To access raw data outside a file system you have to use ioctl's which are low level io calls that get passed directly to device drivers. Typically information is passed in the ioctl with pointers to various system structures and likewise returned. It's a little tougher than regular application level programming.

Seems to me it would be easier to go ahead and use FAT to keep the PC programming simpler and it won't be too hard on the AVR side either because it's been done many times and there are several open source libraries on the net.

Actually I'm not so sure about having to use ioctls to read/write block devices. But probably you should start by opening the device as input instead of create and just get reading to work before moving on to writing/formatting. If you can write it you can zero/format it yourself

This is for linux but windows will be similar - take a look at chapter 6 and appendix B.

Advanced Linux Programming

You are probably right with just using FAT32, but I would just like to be able to get all 8GB from my 8GB card, not 7.6GB. just something I would like to be able to do. I will take a look at the linux stuff when I get a chance, thanks for the lead.

Garret -

On Windows, I'm pretty sure you need to be in Kernel mode to get the kind of access you're talking about. That means having to write a driver for your "User" mode program to go through to access the SD card. That's a tall order. Oddly enough, using the MCU is easy. You can hook up an SD card to it on the SPI lines and just pass through PC commands/data on the UART Tx/Rx lines. There are some links to more information HERE.

Open Source examples, rawcopy.cpp is probably closest to what you are trying to do.

Tools and utilities for Windows

I would still go with FAT and just use a 16/32/64gb card if I needed the extra .4gb storage. Never know, might want to read that card from your cell phone or whatever some day and they all like file systems.

But on the other hand, it is fun to be able to do exactly what you want and I always enjoyed learning/using the less common (and more difficult) programming techniques.

Yes I’m on the side of “if I spend 10 dollars on a SD card I want to be able to use it how I want to”. I don’t let the man tell me how to use it. Thanks for the ‘less common more difficult programming techniques’ that are project specific better.

Noter -

Were you able to extract a rawcopy.cpp from that site? All I got was an .exe (no source code).

Garret -

I think the .4gb is allocated for the FAT32 file system expansion itself. Not much to see there unless you want to explore how things are organized. Most of it is blank.

There is no real mystery to this.

The DOS Function calls are still available as well as several methods from within Windows itself.

You do, of course, have to run with Administrator rights unless you are using a wrapper/library that takes care of the Security for you.

If you are writing directly to storage devices you will have to be very careful, which I am sure that you have considered. A small error in your program could crash/corrupt a legitimate file system that you would prefer to be left intact. (Your boot drive for instance.) Misswriting a single cluster, even sector, can have essentially instantaneous and disastrous results.

Method #1:

    #include <stdlib.h>
    #include <stdio.h>

    int main(void)
    {
        FILE *volume = fopen("\\\\.\\C:", "r");
        if (volume)
        {
            long long offset = 0;  // Sector-aligned offset
            setbuf(volume, NULL);  // Disable buffering

            if (_fseeki64(volume, offset, SEEK_SET) == 0)
            {
                char buf[1024];  // Multiple of sector size
                size_t cb = fread(buf, sizeof(*buf), _countof(buf), volume);

                // Process the data
            }
            fclose(volume);
        }
        return 0;
    }#include <stdlib.h>
    #include <stdio.h>

    int main(void)
    {
        FILE *volume = fopen("\\\\.\\C:", "r");
        if (volume)
        {
            long long offset = 0;  // Sector-aligned offset
            setbuf(volume, NULL);  // Disable buffering

            if (_fseeki64(volume, offset, SEEK_SET) == 0)
            {
                char buf[1024];  // Multiple of sector size
                size_t cb = fread(buf, sizeof(*buf), _countof(buf), volume);

                // Process the data
            }
            fclose(volume);
        }
        return 0;
    }

Method #2:

    LARGET_INTEGER lDiskOffset;

    PDEVICE_OBJECT pDevObj; //Device object representing disk/partition

    KEVENT Event;

    // Trying to read some arbitrary sector number 1169944 and 
    // by default assuming sector size

    // 512

    ..........

    ..........

            lDiskOffset.QuadPart = 1169944*512;
            sBuf = ExAllocatePool(NonPagedPool, size);

            if (!sBuf) {
                ObDereferenceObject(pFileObj);
                return STATUS_INSUFFICIENT_RESOURCES;
            }
            KeInitializeEvent(&Event, NotificationEvent, FALSE);
            memset(sBuf, '0x00', size);
            pIrp = IoBuildSynchronousFsdRequest(IRP_MJ_WRITE/*IRP_MJ_READ*/, 
                pDevObj, sBuf, size, &lDiskOffset, &Event, &ioStatus);

            if (!pIrp) {
                ExFreePool(sBuf);
                return STATUS_INSUFFICIENT_RESOURCES;
            }

            status = IoCallDriver(pDevObj, pIrp);

            if (status == STATUS_PENDING) {
                KeWaitForSingleObject(&Event, Executive, KernelMode, FALSE,    NULL);
                status = ioStatus.Status;
            }
            ExFreePool(sBuf);

    ..........

I think that those 2 examples should be enough for you to find out more.

I would advice, at the very least, to use a partition table that Windows would recognize...

pcbolt - The link for sources is at the top of the page.

Thanks so much Jim I’ve got it now. And I didn’t corrupt my hard drive doing it. I had to change your code a little to get to work on my compiler/ computer (I’m not good enough with C to get into compilers). I noticed that the first sector of the SD card has important data and needs to be left alone. If it’s written over it won’t let you open it next time and a re-format is necessary, I’m sure there is a simple explanation. I can’t wait to use all 8GB for data. Thanks again.

This to write to a sector

#define BUFFER_SIZE 512

// Write to sector //
int main(void)
{
    FILE *volume;
    int k = 0;
    long long sector = 0;
    char buf[BUFFER_SIZE] = {0};

    sector = 5;                 // Sector-aligned offset
    for(k=0;k<BUFFER_SIZE;k++)  // Fill buffer with dummy value
        buf[k] = 'R';

    volume = fopen("\\\\.\\E:", "a");
    setbuf(volume, NULL);       // Disable buffering
    if(!volume)
    {
        printf("Cant open Drive\n");
        return 1;
    }

    if(sector == 0)
    {
        printf("Cant write to that sector I'm not sure why ");
        printf("but if you do it wont work the next time");
        return 2;
    }

    if(fseek(volume, sector*BUFFER_SIZE, SEEK_SET) != 0)
    {
        printf("Cand move to sector\n");
        return 2;
    }

    // Write buffer to sector //
    fwrite(buf, sizeof(*buf), BUFFER_SIZE, volume);

    fclose(volume);

    return 0;
}

And this to read from a sector

#define BUFFER_SIZE 512

// Read from sector //
int main(void)
{
    FILE *volume;
    int k = 0;
    long long sector = 0;
    char buf[BUFFER_SIZE] = {0};

    sector = 30304;             // Sector-aligned offset

    volume = fopen("\\\\.\\E:", "r");
    setbuf(volume, NULL);       // Disable buffering
    if(!volume)
    {
        printf("Cant open Drive\n");
        return 1;
    }

    if(fseek(volume, sector*BUFFER_SIZE, SEEK_SET) != 0)
    {
        printf("Cand move to sector\n");
        return 2;
    }

    // read what is in sector and put in buf //
    fread(buf, sizeof(*buf), BUFFER_SIZE, volume);

    // Print out what wiat in sector //
    for(k=0;k<BUFFER_SIZE;k++)
        printf("%02i ", buf[k]);

    fclose(volume);

    return 0;
}

Jim -

Thanks for that code. Learned something new everyday here.

That is good nothing got corrupted.

People often times see REALLY LARGE HARD DRIVES and don't really realize that a very small amount of data misswritten and it could all be gone... (Well "technically", not gone but rather inaccessible...)

If you go to Wikipedia and look up "Fat32" it will redirect you to "File Allocation Table".

There you will find a good technical description on what the "First Sector" is all about. (That was my previous remark about a Partition Table. It is just easier to keep one that Windows would recognize even if the File System is invalid...)

I usually find it easier, more helpful, to have some sort of file system, but there are times where that isn't really important. Especially since small flash cards are REALLY CHEAP and still provide HUGE STORAGE for a Microcontroller.

Another thing you can do, which you may have thought of, is getting a few of the MicroSD Cards. You can use the MicroSD-to-SD-Card-Adapter as your Flash Socket and solder directly onto the end of it. (I find that helpful/convenient too!)

Glad the code could get in you the proper direction and or is of interest. (To "Both"...)

Now I’m having another problem, when I use the fwrite function if one byte in the buffer is 0x0A (10) it won’t load it correctly. I think its treating this as “NL line feed, newline”. Any ideas how to make fwrite just print the data in the buffer exactly how it is. Thanks.

OK I figured it out here it all about “ab” opens it in binary form.

volume = fopen("\\\\.\\E:", "ab");

SG, have you considered using CLKO to provide an external clock for your slave chip? Even the same freq crystals will drift a bit over time but if both/all chips run off the same clock they will stay in step forever.

I think that post was intended for this thread. Anyways my overall idea is to have 2 completely independent sensors not connected at all. But as I just learned crystals have a stability rating in ppm (parts per million isn’t a just a chemistry thing anymore) and with an error of 100ppm there will be an error of 4.3 minutes per month. My expected run time is about 3 hours resulting in 1.08 sec shift at worst case, which I might just have to live with.

But here is a concept I just thought up, what if there was some way to re connect them after and detect the phase shift. That sounds too complicated for now.

Scooter,

I am curious, what C Compiler are you using for your Windows Programming?

I have CodeBlocks 10.05 I don’t know much about picking compilers but it says ‘GNU GCC’ under the compiler setting so I stick to that.

Wow scootergarrett, Code::Blocks is a fantastic IDE.

It even has the AVR GCC compiler all set to build AVR projects besides doing C and C++.

I am just starting my first attempt so I assume there will be a learning curve, but so far it seems pretty straight forward.

Thanks so much.

Ralph

New micro-SD project is chilling out the window talking temperature data. So I have a set up so I can put a sampling frequency and duration on an SD card pop it in the MCU turn it on and let it run. Then take the card out and extract the data. Data to and from the nerdkit with no wires. The code defiantly needs work, if the duration is set to long bad things are happening. I think the writing to the SD card is causing it to skip AD conversions but if I better work out my interrupts I can probably fix that (testing soon).

Anyways I have the whole circuit on 3.3 Volts to better accommodate the SD card. Powered by a little 6V Duracell 28L which might be my new battery of choice for small projects that still need to last (250mAhr also needs testing). Circuit diagram Circuit lab.

And MC code so far:

// SDHC Card test program
// scootergarrett
// for 328p

#define F_CPU 14745600

#include <stdio.h>
//#include <math.h>
#include <string.h>

#include <avr/io.h>
#include <avr/interrupt.h>
#include <avr/pgmspace.h>
#include <inttypes.h>

#include "../libnerdkits/io_328p.h"
#include "../libnerdkits/delay.h"
#include "../libnerdkits/uart.h"
#include "SDHC.h"

volatile uint8_t OverFlowFlag = 0;

uint16_t adc_read(int port)
{
    ADMUX = port;
    ADCSRA |= (1<<ADSC);
    while(ADCSRA&(1<<ADSC));
    return (ADCL+(ADCH<<8));
}

int main()
{
    int k = 0;
    uint32_t CurrentSector = 2;
    uint32_t LastSector = 25;

  //  uint16_t CurrentMeasurement = 0;

    DDRC |= (1<<PC3);               // Make PC3 an output for testing
    PORTC  &= ~(1<<PC3);

    // start up the Analog to Digital Converter //
    ADMUX = 0;
    ADCSRA = (1<<ADEN) | (1<<ADPS2) | (1<<ADPS1) | (1<<ADPS0);
    ADCSRA |= (1<<ADSC);

    // start up the serial port //
    uart_init();
    FILE uart_stream = FDEV_SETUP_STREAM(uart_putchar, uart_getchar, _FDEV_SETUP_RW);
    stdin = stdout = &uart_stream;

    // Start the SDHC //
    spi_init();
    sdhc_init();
    delay_ms(100);

    // Read the second sector for inportnat data //
    sdhc_read_block(2);
    CurrentSector = *(uint32_t*)&buffer[0x10];
    LastSector = *(uint32_t*)&buffer[0x20];

//    printf_P(PSTR("CurrentSector: %ld\n\r"), CurrentSector);
//    printf_P(PSTR("LastSector: %ld\n\r"), LastSector);
//    printf_P(PSTR("OCR1A from buffer: %i\n\r"), *(int16_t*)&buffer[0]);

    // Interrupt setup stuff //
    TCCR1B |= (1<<CS12) | (1<<CS10) | (1<<WGM12);
    TIMSK1 |= (1<<OCIE1A);
    OCR1A = *(int16_t*)&buffer[0];                 //Interrupt timming
    sei();

    while(1)
    {
        if(OverFlowFlag)
        {
          //  CurrentMeasurement = adc_read(0);

            *((uint16_t*)&buffer[k*2]) = adc_read(0);
            ++k;

        //    printf_P(PSTR("%04i\n\r"), CurrentMeasurement);

            if(k*2>BUFFER_SIZE)
            {
                sdhc_write_block(CurrentSector);
                ++CurrentSector;
                k = 0;
            }

            if(CurrentSector == LastSector)     // Never writes to last sector
                break;

            PORTC  ^= (1<<PC3);     // Change LED each sample
            OverFlowFlag = 0;
        }
    }

    while(1)                    // Done Loop
    {
        PORTC  ^= (1<<PC3);     // Blink LED for at end
        delay_ms(500);
    }

    while(1);
    return 0;
}

ISR(TIMER1_COMPA_vect)
{
    OverFlowFlag++;

    return;
}

Will post the data extract code if anyone cares

Out window plot running a few minutes Check out that first order response

SG -

I see where you set the frequency

OCR1A = *(int16_t*)&buffer[0];

but I didn't see where the duration is set. I do know the writes to the SDHC card are pretty slow (in MCU time) so you could try increasing the SPI clock frequency or maybe just disable the timer interrupt while your writing to the card. You can also set the ADC to operate in "Auto Trigger Mode" when the timer hits OCR1A value, then have the the ADC "on complete" interrupt store the results and update the buffer pointer. Then all your "main" code has to do is check the buffer pointer (if it's a global variable) and write to the card when it's full. If you're worried about battery life you can also look into the various sleep modes...but I guess one thing at a time, eh?

The durations is set by data already on the SD card. When I initialize the SD card I determine the Last sector that will be used, so then I just need to break out of the cycle if the current sector is == to the last sector.

So for running faster I changed the code so the buffer is getting fill constantly every interrupt and the SD card is getting written when the buffer is full, so as long as the SD card get the first couple of numbers in the buffer before the next interrupt starts filling the buffer again there wont be a problem. One issue now is that my computer can’t malloc data for 11520000 points (8hr at 400 Hz). It will statically allocate that much memory but then numbers with in the program need to be change, I like the ability to pop the card in and have the program malloc the correct amount of data (thinking of non computer people using the sensor).

I also added a sleep mode to the end so when the sampling time is over it will save battery life.

CODE:

// SDHC Card test program
// scootergarrett
// for 328p

#define F_CPU 14745600

#include <stdio.h>
//#include <math.h>
#include <string.h>

#include <avr/io.h>
#include <avr/interrupt.h>
#include <avr/pgmspace.h>
#include <inttypes.h>
#include <avr/sleep.h>

#include "../libnerdkits/io_328p.h"
#include "../libnerdkits/delay.h"
#include "../libnerdkits/uart.h"
#include "SDHC.h"

volatile k = 0;

uint16_t adc_read(int port)
{
    ADMUX = port;
    ADCSRA |= (1<<ADSC);
    while(ADCSRA&(1<<ADSC));
    return (ADCL+(ADCH<<8));
}

int main()
{
    uint32_t CurrentSector = 2;
    uint32_t LastSector = 25;

  //  uint16_t CurrentMeasurement = 0;

    DDRC |= (1<<PC3);               // Make PC3 an output for testing
    PORTC  &= ~(1<<PC3);

    // start up the Analog to Digital Converter //
    ADMUX = 0;
    ADCSRA = (1<<ADEN) | (1<<ADPS2) | (1<<ADPS1) | (1<<ADPS0);
    ADCSRA |= (1<<ADSC);

    // start up the serial port //
    uart_init();
    FILE uart_stream = FDEV_SETUP_STREAM(uart_putchar, uart_getchar, _FDEV_SETUP_RW);
    stdin = stdout = &uart_stream;

    // Start the SDHC //
    spi_init();
    sdhc_init();
    delay_ms(100);

    // Read the second sector for inportnat data //
    sdhc_read_block(2);
    CurrentSector = *(uint32_t*)&buffer[0x10];
    LastSector = *(uint32_t*)&buffer[0x20];

//    printf_P(PSTR("CurrentSector: %ld\n\r"), CurrentSector);
//    printf_P(PSTR("LastSector: %ld\n\r"), LastSector);
//    printf_P(PSTR("OCR1A from buffer: %i\n\r"), *(int16_t*)&buffer[0]);

    // Interrupt setup stuff //
    TCCR1B |= (1<<CS12) | (1<<CS10) | (1<<WGM12);
    TIMSK1 |= (1<<OCIE1A);
    OCR1A = *(int16_t*)&buffer[0];                 //Interrupt timming
    sei();

    k = 0;
    while(1)
    {

        if(k*2>BUFFER_SIZE)
        {
            k = 0;
            sdhc_write_block(CurrentSector);
            ++CurrentSector;
            PORTC  ^= (1<<PC3);                 // Change LED each sector
            if(CurrentSector == LastSector)     // Never writes to last sector
                break;
        }
    }

    cli();                      // Turn off interupts
    k = 0;
    PORTC  &= ~(1<<PC3);        // Turn off light

    PRR = (1<<PRTWI)            // turn off TWI
         | (1<<PRTIM2)          // turn off Timer/Counter 2
         | (1<<PRTIM0)          // turn off Timer/Counter 0
         | (1<<PRTIM1)          // turn off Timer/counter 1
         | (1<<PRSPI)           // turn off SPI
         | (1<<PRUSART0)        // turn off USART (will turn on again when reset)
         | (1<<PRADC)           // turn off ADC*/
        ;

    set_sleep_mode(SLEEP_MODE_PWR_SAVE);
    sleep_mode();

    while(1);
    return 0;
}

ISR(TIMER1_COMPA_vect)
{
    *((uint16_t*)&buffer[k*2]) = adc_read(0);
    ++k;

    return;
}

So when I moved to taking 2 measurements back to back I got some strange results. This is with 2 identical temperature sensors. There is one bit noise half the time it takes to fill a buffer, any ideas what could cause this? Could communicating with the SD card be affecting measurements Interrupt code for 2 measurements at a time:

ISR(TIMER1_COMPA_vect)
{
    ADMUX = 0;
    ADCSRA |= (1<<ADSC);
    while(ADCSRA&(1<<ADSC));

    ADMUX = 1;
    ADCSRA |= (1<<ADSC);
    *((uint16_t*)&buffer[k]) = (ADCL+(ADCH<<8));

    while(ADCSRA&(1<<ADSC));
    *((uint16_t*)&buffer[k + 2]) = (ADCL+(ADCH<<8));

    k += 4;

    return;
}