Daniel Campos wrote:

# Network and XML

- Introduction
  - Think that...
  - So...
  - XML, RPC and SOAP
  - Gambas must be there
  - Network component
- Programming with "Net" component
  - Connected Sockets: Clients and Servers
    - The Client Side : Acting like a TCP or Local (Unix) Client
    - The Server Side : Acting like a TCP or Local (Unix) Server

## Introduction

The future, al least for the next years, is based in networks and standards.

Even big companies that always tried to hide all its internal file formats, have changed partially its mind, and now it seems they have discovered the fire, talking about XML, SOAP, RPC, and things like this, that probably, could be usual stuff today if they were put some effort and money on them in the past.

### Think that...

No one can imagine isolated devices today : you have to extract data from them, know what's happening even if you are in the other side of the world.

Devices are made by different manufacturers, and costumers think it's not their matter to make them compatible. So, companies need standars to share information.

Companies need to control its money, information is power and a chance to grow and earn new money. Companies trust in the support of big hardware and software companies.

### So...

New back-office systems provided by main companies, are based on known standards, mainly : data bases (SQL), http servers like Apache, XML, RPC and SOAP. If big companies acts in that way, medium and small companies will do the same...or die.

### XML, RPC and SOAP

When somebody is learning XML, the first idea is "what an stupid thing". Well, XML seems to be only tags and random data you put in a file. And XML is really that. The power of XML is that is an international standard, and that can be easily used by people and computers. Lots of people can learn it, not just programmers, create an implementation, write a XML document with a text editor, and lot of libraries and other software will help them to share their documents, or translate it to HTML or PDF beautiful reports.

And HTTP is just a protocol, there are lots of them. Protocols are what languages are for people, if two of more people know one language, they can share information. But the birth of Arpanet and the grow of Internet made this protocol quite popular: it is used to receive web pages, images, files, commercial data and all the stuff you can imagine from servers that are placed all around this world. So, almost all network hardware and software (routers, firewalls, end-user computers, servers...) is ready to manage this protocol. Think it twice: If you use HTTP to send and receive your information, you don't need to worry about the physical ubication of your target : all the hardware in the whole world, a big net of subnets, will carry the information without asking you any question, just press the right button (well, usually the left button of mouse) , and let the big machine work.

Here's where we find XML-RPC and SOAP. They are protocols to allow computers to talk between them. Using this protocol, they can do more than sharing some documents: one computer can send orders to other computer to perform some task, and then the second computer will return the result of the action. You can find this into your computer : some programs call to libraries and other programs to perform its work. The only new thing in XML-RPC and SOAP, is that now, conversations are not only into your computer, but between computers.

But this is a big thing. The idea is not new at all, but now, it is based on network and data format standards: To carry the information HTTP, to write orders, questions and answers, XML. The end of this is that today, it is quite easy for programmers and even for some users, to write programs that will be connected with other programs running in other computers, in different parts of the world. And the other important point, is that, you don't need to have a defined hardware-software platform, they are standards : you can both buy hardware and software from many different providers. You can buy what you need : low price? high performance? scalability?, just look, compare, and buy or even use freely the best thing for you.

### Gambas must be there

Gambas needs to implement standars, and I'd say more : Linux needs Gambas. Gambas is a BASIC language interpreter. BASIC is an easy language, anybody can learn a little, and start making little programs and scripts. A lot of programmers started learning BASIC when they were children, may be on 8-bits CPUs. A lot of people working in different sections of different companies, knows at least a little of BASIC language. They are not programmers at all, but they have what they need as they can write macros to help them on his work with little and medium data bases, spread-sheets and text documents. They know how to create a formulary, write and read data from tables, share some information sending it by mail... By other side, there are tons of web depelopers, too busy to learn about pointers and memory management, they have to write HTML, create beautiful pictures, administrate servers, deal with hackers and many other things. They need a simple language to create CGI's. There are things like PHP or Perl, but there's one called ASP which is not more than BASIC mixed with HTML, so they know BASIC.

So, the conclusion is that lots of people are waiting for BASIC to migrate to Linux. What can we do here? We are in the middle of the road : we have a simple and well-designed GUI (not very much projects can say that), we have the interpreter, and we have lots of libraries written in C or C++ waiting to help us, may be the biggest API that any operating system has ever known. To cross to the other side we need only to implement that standars that will allow programmers and users to connect to the information they need.

Network component is a piece in that target: Let's open the door that will give us the rest of the world.

### Network component

It is not really a component, it will be implemented in (al least) three components: "net" "net.curl" and "gbxml".

"Net" is the base of all:

Currently the web is based on TCP/IP protocols. We can say that the IP part of this protocol is matter of operating system, we have to care about "TCP/" part. To implement all stuff we have the following classes:

- DnsClient : is a class to convert host names (like "gambas.sourceforge.net"), which are understood by humans to IP addresses (like 192.168.0.1) which are the real names for computers. It can perform its work using operating systems resources: DNS, NIS, hosts file,NMB lookup, LDAP, etc...
- Socket : to share information with servers, we need to stablish a TCP connection. This class can do that, and send and receive data in "raw" format.
- ServerSocket : sometimes we'll be the servers, and we need to let clients connect with us. This is the class to do it using TCP/IP.
- UdpSocket : some internet tasks, like transfering multimedia streaming data, do not work fine if they use a connection, so they usually use UDP, a more "primitive" way of transfering data, with no flow control. Datagram class is intended to create both UDP clients and servers.
- LOCAL or UNIX sockets : sometimes we don't need to connect with other computers, but with other programs in the same computer. Using TCP for this is not efficient, and you waste a limited resource as TCP ports are. Unix sockets are local sockets implemented by the operating system to "emulate" TCP sockets into the computer ( oh, well, this is not a technical conference, I simply try to give an idea, you know? ), and data sharing using this method is quite fast and inexpensive. Both ClientSocket and ServerSocket can also stablish unix-socket connections.
- SerialPort : Now let's speak about the rest of computing world. Not all devices, specially designed for industrial or commerce tasks have an ethernet connection. May be 70% of them have RS-232, RS-485, or RS-422 serial ports. And programmers often have to stract or send information to them, and then compute the information and convert it to data to be placed in a database, web pages, and so on. So, there's also a class called "SerialPort" to allow send and receive data using a serial port.
- SerialPort, Socket and UdpSocket classes inherits from generic ".Stream" class. This means you can use them as file descriptors,usig standard Gambas methods like "read", "write", or "close", so you will find easy to learn using them.

"Net.curl"

'libcurl' is a free and portable library written by some people at http://curl.haxx.se. It provides all the pieces needed to manage high level network protocols, like HTTP, FTP and TELNET. This library will be useful to write a component with the following classes:

- HttpClient : by using ClientSocket and ServerSocket you'll receive only raw data, that you have to manage and verify. This client allows you to connect with HTTP servers (like Apache, yes), managing all HTTP protocol, you have just to care about the document (or object, as DOM people like to say) you receive. (this class is in ALPHA state)

- FtpClient : FTP protocol is one of the basic tools for any company: they can send and receive data, for example in ASCII format, and then proccess it to consolidate data in a database. (not yet implemented)

- TelnetClient : Allowing GAMBAS programs to remote control other machines. (not yet implemented)

"GBXML" are the tools to connect with the world: (GBXML is not yet implemented)

- There will be at least two different pure-XML classes:
  - A tree based class : it saves all XML information in memory, which makes work with XML very easy, you can move to any part of the document, and modify it easily, and using very few time.
  - An XMLReader class, more conventional : it just reads pieces of XML document from a file, without keeping it in memory. May be it is more slow, but when you have a big document, a tree based model can use all your memory resources, even hanging the computer.
  - May be a SAX based XML class : it is useful to parse documents while you are receiving it from a stream. (it is useful for other things, too)
- There will be XML-RPC client class : it will allow you to call remote RPC-servers and receive data just as if you were calling methods from your own program, or at least in a quite similar way. Nothing about TCP/IP or XML, just Integers, Strings, Arrays...
- There must be also a XML-RPC server, for two reasons : the first, to "decrypt" incoming information from client in XML format to data types, and second, to allow create stand-alone servers without the need of "monsters" like Apache.

In the future there will be work on SOAP front. The idea of SOAP is quite similar to XML-RPC, it is also based on XML and HTTP, but with SOAP you can perform very complex tasks, it is powerful, but by the other side is quite complex.

## Programming with "Net" component

### Connected Sockets: Clients and Servers

A socket is just a place from where you can read and write data. However, socket definition does not speak about servers, clients, connections, etc, this is a concept placed in the next floor. TCP and Local sockets are particular socket implementations in which there are flow control and specifications about the program roles :client and server.

There are two ways to work with connected sockets:

- TCP sockets : you can use them to communicate with remote or local programs
- Local or UNIX sockets : just to communicate with programs in your own machine

Now let's implement two programs to play with TCP and Local sockets:

- The client will connect with server and it will send an string 'hello' to the server.
- The server will return 'bye'.
- The client will close the connection after bye message reading.

#### The Client Side : Acting like a TCP or Local (Unix) Client

To create a program in which you will connect to a remote or local server using TCP sockets or Local sockets, you have to use "Socket" class. We will use here two ways to implement that, the first implementation does not use any event:

- 1) Create a new project called "Sock1".
- 2) Go to "project","properties","components", remove "qt" and add "net", then press "OK"
- 3) Add a new module called M1.
- 4) Edit that code:

- Before Main() method, lets define an object from 'Socket' class:

     ' Gambas module file
    PUBLIC MySock AS Socket
    ...

- Now, the first thing we do is initialize the object:

    ...
     PUBLIC SUB Main()
      MySock=NEW Socket
      ....

- Then, we have to connect with remote or local server, so we need to use Connect() method. If we want to connect using TCP protocol, we need to specify remote host name or ip, and port to connect to:

       ...
       MySock.Connect ( "name_of_host",3450 )
       ...

- If we need to connect to a Local server (UNIX sockets), we have to specify path from that socket:

       ...
       MySock.Connect ("/path/to/socket")
       ...

- Connection proccess takes a time, so we can not start sending or receiving data in the next line of code, instead, we have to wait until connection is finished or there's an error trying to connect. If we really connect, "Status" property from Socket will change to value Socket.Connected (value 7), but if connection process fails, Status property will change to a value less than zero. So, we'll wait in a loop...

    ...
    Do While (MySock.Status <> 7) And (MySock.Status >0 )
       Wait(0.1)
    Loop
    ...

- Note that you must use "Wait" method, to let the event loop update Socket Status.
- After that, let's indicate an error code, something failed, or, if connection was stablished, let's send some data:

     ...
     If MySock.Status<>7 Then
       PRINT "Error"
       Quit
     End if
     Write #MySock, "hello",5
    ...

- Now, let's read reply from server. First, we wait until there's data to read, checking it with Lof method. Then, we will read data:

    ...
    Do While Lof(MySock)=0
       Wait(0.1)
    Loop
    Read #MySock, sBuf, Lof(MySock)
    Print sBuf
    ...

- Finally, we close the socket:

    ....
    CLOSE #MySock

- That's all.Full code:

    PUBLIC MySock AS Socket

    PUBLIC SUB Main()

      DIM sBuf AS String
      MySock=NEW Socket

      MySock.Connect ( "name_of_host",3450 )

      DO WHILE (MySock.Status <> 7) AND (MySock.Status >0 )
        Wait(0.1)
      LOOP
       IF MySock.Status<>7 THEN
        PRINT "Error"
        QUIT
      END IF

      WRITE #MySock, "hello",5

      DO WHILE Lof(MySock)=0
        Wait(0.1)
      LOOP

      READ #MySock, sBuf, Lof(MySock)
      PRINT sBuf
      CLOSE #MySock

    END

- Now, let's implement another version, using events:

- 1) Create a new project called "Sock2".
- 2) Go to "project","properties","components", remove "qt" and add "net", then press "OK"
- 3) Add a new class called "ClsMain".
- 4) Edit that code:

- We need an instance of our class, and a Socket :

        ' Gambas class file
        STATIC App AS ClsMain
        PUBLIC  MySock AS Socket
        ....

- At _New method from our class, we'll start socket connection:

    ...
    PUBLIC SUB _New()
      MySock=NEW Socket AS "MySock"
      MySock.Connect("name_of_host",3450)
    END
    ...

- At main method, we create a new instance of our class, so we start connection, as _New method is called when creating an object :

    ...
    STATIC PUBLIC SUB Main()
      App=NEW ClsMain
    END
    ...

- When connection has been stablished successfully, "Ready" event from socket raises, and we send our string to server :

    ...
    PUBLIC SUB MySock_Ready()
        WRITE #MySock,"Hello",5
    END
    ...

- But, if there was an error, "Error" event raises :

    ...
    PUBLIC SUB MySock_Error()
        PRINT "Unable to connect"
    END
    ...

- When new data arrives, "Read" event raises, so we read that data, put it on screen, and close the socket :

    ...
    PUBLIC SUB MySock_Read()
      DIM sCad AS String
      READ #MySock,sCad,Lof(MySock)
      PRINT sCad
      CLOSE #MySock
    END
    ....

- Full code :

    ' Gambas class file
    STATIC App AS ClsMain
    PUBLIC  MySock AS Socket

    PUBLIC SUB MySock_Ready()

      WRITE #MySock,"Hello",5

    END

    PUBLIC SUB MySock_Read()

      DIM sCad AS String
      READ #MySock,sCad,Lof(MySock)
      PRINT sCad
      CLOSE #MySock

    END

    PUBLIC SUB _New()

      MySock=NEW Socket AS "MySock"
      MySock.Connect("name_of_host",3450)

    END

    PUBLIC SUB MySock_Error()

       PRINT "Unable to connect"

    END

    STATIC PUBLIC SUB Main()

      App=NEW ClsMain

    END

#### The Server Side : Acting like a TCP or Local (Unix) Server

To work a a server, we need 'ServerSocket' class. It listen for connections, and returns a new Socket object for each client connection, so we can manage multiple clients.

- 1) Create a new project called "Srv".
- 2) Go to "project","properties","components", remove "qt" and add "net", then press "OK"
- 3) Add a new class called "ClsServer".
- 4) Edit that code:

- As we did in the second client example, we need an instance of our class. We also need a ServerSocket object, and an array of objects to place the Socket objects that we need to communicate with our clients

    STATIC Server AS ClsServer
    PUBLIC Clients AS Object[]
    PUBLIC Srv AS ServerSocket
    ...

- At program start, we create the instance of our class

    ...
    STATIC PUBLIC SUB Main()
      Server=NEW ClsServer
    END
    ...

- When class is loaded by the interpreter, it calls to _New method. We initialize here the array of objects, and start the server. To do that, we have to specify TCP port to listen to, type of socket, and call to 'Listen' method

    ...
    PUBLIC SUB _New()
      Clients =NEW Object[]
      Srv=NEW ServerSocket AS "Srv"
      Srv.Port=3450
      Srv.Type=ServerSocket.Internet
      Srv.Listen()
    END
    ...

- If we'd like to use Local or Unix sockets, instead of TCP sockets, we should specify Local socket type, and a path instead of a port

    ...
    PUBLIC SUB _New()
      Clients =NEW Object[]
      Srv=NEW ServerSocket AS "Srv"
      Srv.Path="/path/to/my/socket"
      Srv.Type=ServerSocket.Local
      Srv.Listen()
    END
    ...

- When server is listening, each time a client tries to connect to our service, 'Connection' event from server raises. Here we must accept that connection by calling 'Accept' method. It returns to us a 'Socket' object to manage that connection, so there will be a 'Socket' object for each client connection, that we store in our object array

    ...
    PUBLIC SUB Srv_Connection(Host AS String)
      DIM MySock AS Socket
      PRINT "Accepting connection from --> " & Host
      MySock=Srv.Accept()
      Clients.Add(MySock)
    END
    ...

- Our client will send us 'hello' message, and then 'Read' event from 'Socket' will raise. Note that we use here 'LAST' keyword to know which of our clients has sent that message. Here we read that string, and send our string 'bye'

    ...
    PUBLIC SUB Socket_Read()
      DIM sCad AS String
      READ #LAST,sCad,Lof(LAST)
      PRINT "Received data -->" & sCad
      WRITE #LAST,"bye",3
    END
    ...

- Finally, after client has closed the conection, we receive 'Closed' event, and clear that 'Socket' object from our array, as that connection is alive no more

    ...
    PUBLIC SUB Socket_Closed()
      PRINT "Connection closed"
      Clients.Remove(Clients.Find(LAST))
    END
    ...

- Full code:

    ' Gambas class file
    STATIC Server AS ClsServer
    PUBLIC Clients AS Object[]
    PUBLIC Srv AS ServerSocket

    PUBLIC SUB Socket_Read()

      DIM sCad AS String
      READ #LAST,sCad,Lof(LAST)
      PRINT "Received data -->" & sCad
      WRITE #LAST,"bye",3

    END

    PUBLIC SUB Socket_Closed()

      PRINT "Connection closed"
      Clients.Remove(Clients.Find(LAST))

    END

    PUBLIC SUB Srv_Connection(Host AS String)

      DIM MySock AS Socket
      PRINT "Accepting connection from --> " & Host
      MySock=Srv.Accept()
      Clients.Add(MySock)

    END

    PUBLIC SUB _New()

      Clients =NEW Object[]
      Srv=NEW ServerSocket AS "Srv"
      Srv.Port=3450
      Srv.Type=ServerSocket.Internet
      Srv.Listen()

    END

    STATIC PUBLIC SUB Main()

      Server=NEW ClsServer

    END