Saturday, 12 August 2023

ESP8266Webserver and POST requests

Good examples of the Arduino ESP8266Webserver are difficult to find when it comes to writing a HTTP server which accepts POST.

There are different ways a POST request can commonly be made:

  • Using a form
  • Performing a simple fetch in Javascript
  • Performing a multipart fetch in Javascript

And there are different ways the server should respond depending on how the submission is made.

The Request

Using a form

<form action="/upload?o=option" method="POST" enctype="multipart/form-data">
    <p>File: <input type="file" name="name"></p>
    <input type="submit" value="Upload">
</form>

Performing a multipart Fetch in Javascript

Performing a multipart fetch in Javascript requires the use of a FormData() object, which is populated with the files.  For each entry that is appended to the form, a label, a binary object containing the file, and a filename are supplied.

The webserver is unable to access the Mime Type for the each supplied object (e.g. application/octet-stream, so the Mime Type must be inferred from the filename or the label).  Additional information can be passed with a query parameter - e.g. ?o=option if needed.

// Prepare binary and text data
const binarydata = new Uint8Array(new ArrayBuffer(1024)) ;
const textdata = "textmessage" ;
compressText(textdata, binarydata) ;

// Create formData object
const formData = new FormData() ;

// Store binary data
const b_blob = new Blob([binarydata], { type: "application/octet-stream"}) ;
formData.append("binarylabel", b_blob, "binary.dat") ;

// Store text data
const t_blob = new Blob([textdata], { type: "text/plain" }) ;
formData.append("textlabel", t_blob, "text.txt") ;

// Post the message
const request = new XMLHttpRequest() ;
request.open("POST", "/upload?o=option") ;
request.send(formData) ;

Performing a Simple Request in Javascript

Simple requests can be made, but only readable text can be supplied.

fetch("/upload?o=option", {
  method: "POST",
  body: JSON.stringify({
    "id": 7,
    "complete": true
  }),
  headers: {
    "Content-type": "application/json"
  }
});

ESP8266Webserver on

The ESP8266Webserver has an 'on' function, which can be used to register handlers for different uris.  There are several forms the 'on' function can take, and two of them are relevant.

The on function can be supplied with three callbacks, which the documentation calls 'fn' and 'ufn'.  'ufn' represents the upload function. so one may think that this is always used, however that is not the case.  In fact, 'ufn' is only used with multipart and form posts, and simple requests actually use the 'fn' callback.
  1. void on(const Uri &uri, THandlerFunction handler);
  2. void on(const Uri &uri, HTTPMethod method, THandlerFunction fn);
  3. void on(const Uri &uri, HTTPMethod method, THandlerFunction fn, THandlerFunction ufn);

void on(const Uri &uri, THandlerFunction handler)

The first callback simply maps to the second callback with the method fixed to HTTP_ANY, which allows any of the methods to be used


void on(const Uri &uri, HTTPMethod method, THandlerFunction fn)

The second callback registers a uri along with a method  and provides a callback function, for example:

Webserver.on("/get", HTTP_GET, myGetFunction) 

In the above example callback is made with no reference to any class object, and if declared within a class, it must be declared as a static function (see lambda function discussion below for an alternative).

class WebServer::ESP8266WebServer {

    WebServer() {
        on("/handle", HTTP_ANY, WebServer::doHandle) ;
    }

    static void doHandle() {
      Server.doProcess() ;
    }

    void doProcess() {
     switch (method()) {
       case HTTP_GET:
          send(200) ;
          break ;
       case HTTP_POST:
          // body of the post is treated specially
          // note that this is a simple mechanism
          // and can only handle text
          String postcontents = arg("plain") ;
          send(200) ;
          break ;
       default:
          send(500) ;
          break ; 
  }

}

void on(const Uri &uri, HTTPMethod method, THandlerFunction fn, THandlerFunction ufn)

The third callback is by far the most complex, and includes an upload function which is called to process each file within a multi-part upload (i.e. uploads which have a Content-Type of multipart/*).

If the upload is not multi-part, then the standard fn is called.

This function can be used with the HTTP_ANY method (unlike the ESP32 version of the webserver, the method is not a bitfield, and HTTP_GET | HTTP_POST is not allowed).

In this example, lambda functions [&]() are used to retain the context in the callback.

  on(
    "/config",
    HTTP_ANY,
    [&]() {
      switch (method()) {
        case HTTP_GET:
          serveConfigFile() ;
          break ;
        case HTTP_POST:
          send(400, "text/plain", "Requires multipart/* Content-Type");
          break ;
        default:
          send(405, "text/plain", "Method not supported");
          break ;
      }
    },
    [&]() {
      handleConfigUpload() ;
    }
  );


Disadvantages of the 'fn' callback:
  1. Binary uploads are not supported
  2. The text upload is stored in a string, and the user has no control over memory allocations

Upload Callback Function

The upload callback function is called several times for each file within a multi-part attachment.  For each part, the upload moves through several states:

UPLOAD_FILE_START:

In this state, the handler needs to prepare to receive the incoming file.
This could mean allocating memory, or opening a file handle ready to receive  the file data.
It is not clear as to whether there could be two concurrent uploads - one in the FILE_WRITE state and another in the FILE_START state.

Details of the transaction can be obtained through the upload function:

upload().filename - name of the file as supplied on the form / in the javascript
upload().name - label of the entry as supplied on the form / in the javascript
upload().type - would have expected this to be the mime type of the object, but it is currently random rubbish (12/08/2023)

UPLOAD_FILE_WRITE:

In this state, data is received from the upload (a separate call for each chunk of data, so a large file may have multiple calls).
Here, the handler needs to capture and store the received data.

UPLOAD_FILE_END:

In this state, the upload is complete.
Here, the handler needs to close, validate and store the uploaded file.


Saturday, 20 May 2023

Getting going with the Ubuntu, the Arduino Development Tools and an ESP8266 Wifi Board

Requirements

  • ESP8266 board with CH340 serial interface (allows USB connection without programmer)
ot
  • ESP8266 board without CH340 serial interface
  • Programming Cable / Jumpers
  • Linux Compatible Serial Interface Adaptor with 3.3V output - e.g. CP2102 compatible DSD TECH USB to TTL Converter

Step 1: Download

Download Arduino IDE 2 AppImage and launch - the version I ended up using was:

# wget https://downloads.arduino.cc/arduino-ide/arduino-ide_2.1.0_Linux_64bit.AppImage

Or download the previous 1.8.19 version:

Unpack as root in /opt/arduino-1.8.19

Run /opt/arduino-1.8.19/install.sh as root

Run /opt/arduino-1.8.19/linux-user-install.sh <username>


Step 2a: Launch IDE V2

Change the permissions so that the AppImage is executable, and launch it:

# chmod u+x arduino-ide_2.1.0_Linux_64bit.AppImage

# ./arduino-ide_2.1.0_Linux_64bit.AppImage


Step 2b: Launch IDE V1

Launch the IDE from the start menu

Step 3: Install ESP8266 Board Info

Select File/Preferences and add the following URL to the "Additional Board Manager URLS" and select OK.

http://arduino.esp8266.com/stable/package_esp8266com_index.json

Launch Board Manager and search for "Generic" and install the "esp8266" board.

When it appears in Board Manager, select the desired version (I'm using the latest 3.12) and Install

In the board select drop-down, pick "Generic ESP8266"


Step 4: Select the board

In a new design, select Tools/Board/ESP8266 Boards/NodeMCU (ESP 12E Module)


Step3b: Install ESP8266 Board Info from Git

https://arduino-esp8266.readthedocs.io/en/3.0.0/installing.html

mkdir ~/Arduino/hardware

cd ~/Arduino/hardware

git clone https://github.com/esp8266/Arduino.git esp8266

Step 4: Add some Code

Select some code (e.g. https://randomnerdtutorials.com/esp8266-web-server/) and paste into the main edit window


Step 5: Plug in the Device to the USB Port

I found that the brltty driver was taking over the serial device and needed disabling

# dmesg | tail

159425.909714] ch341 3-1:1.0: ch341-uart converter detected
[159425.910391] ch341-uart ttyUSB0: break control not supported, using simulated break
[159425.910515] usb 3-1: ch341-uart converter now attached to ttyUSB0
[159426.484288] input: BRLTTY 6.4 Linux Screen Driver Keyboard as /devices/virtual/input/input22
[159426.604333] usb 3-1: usbfs: interface 0 claimed by ch341 while 'brltty' sets config #1
[159426.604766] ch341-uart ttyUSB0: ch341-uart converter now disconnected from ttyUSB0
[159426.604796] ch341 3-1:1.0: device disconnected


# systemctl stop brltty-udev.service

# sudo systemctl mask brltty-udev.service

# systemctl stop brltty.service

# systemctl disable brltty.service

Following this, unplugging and re-plugging the board allows /dev/ttyUSB0 to be created.

# ls -la /dev/ttyUSB0

crw-rw---- 1 root dialout 188, 0 May 20 16:07 /dev/ttyUSB0

If the current user does not have the correct access rights, he won't be able to access the device - to fix this, the user needs to be added to the appropriate group (in this case 'dialout')

# usermod -a -G dialout username

Then the current user must logout and log back in again to ensure that the new group membership is established.

Step 6: Download the Code


Step 7: Add some Tools

Add the "Arduino ESP8266 filesystem uploader" tool from  https://github.com/esp8266/arduino-esp8266fs-plugin

Or the "ESP8266 LittleFS Data Upload" tool from https://github.com/earlephilhower/arduino-esp8266littlefs-plugin/releases

Unpack into ~/Arduino/hardware/esp8266/tools


Step 8: Check Dependencies