Data Extractions - WWW and PICK:
Website Design and Data Extractions

By Dave Johnstone



PixieWeb - A smart Telnet connector for PICK and DMS systems (e.g. D3/U2/ADP/IDS)

PixieWeb is a toolkit for moving data in and out of telnet server datastores. The core of this product is our COM connectivity object. This "socket with attitude" connects to the main telnet TCP/IP port(s) of a PICK (MV-Server) in a manner similar to terminal emulators. So you get the reliability, speed and familiarity of that rock-solid time-honored connection method, enhanced by a command set specifically designed to process and manipulate MV-style commands (TCL, English) and data. It is an end-user friendly middleware connection tool, that transmits source data in a format that can be used by common Windows tools.

Use the PixieWeb DLL for extracting data from, or placing data into, most PICK servers. But Since it is not possible to directly execute its DLL (it requires an EXE for the operating system to load it through its entry point). If there is not a host application to run PixieWeb, then we use as the host (see below).

PixieWeb is a general purpose connectivity device operating between Windows and many text-based systems. It is an ActiveX .dll providing access to the text-system via Telnet and is a general connectivity alternative for ODBC. Clients can be browsers, VB (or other COM aware languages), as well as Windows desk-top applications (VBA). It includes a special command-set to extract formatted data from a text-style application (e.g. PICK) to the user-agent. The commands understand the PICK database multi-value structure. Go here to view a live code sample.

Benefits include:

  • Output of simple and consistent data from terminal-style PICK/DMS programs by screen-scraping
  • A tool-kit for PICK data access, which can be used to solve a wide variety of common data access problems.
  • A low-cost, reliable and high-performance universal socket-based data connection component.
  • Get's more from your data by feeding: reports, spreadsheets, and data analysis produced by the best tools available.

PixieRobot

A data extraction processor from the Web or PICK

PixieRobot, is our programmable (scriptable) software utility, that can perform routine web-scraping or screen-scraping tasks on command. PixieRobot operates automatically in an unattended mode with precision, against most forms of targeted web-sites or PICK servers. PixieRobot will run on most Windows platforms.

PixieRobot is a web robot utility, which is a program that:

  • Visits a web-site
  • Reads the wep-pages
  • Inspects the HTML
  • Extracts or submits forms

OR

  • Visits a PICK telnet server application
  • Reads a query or screen
  • Inspects the data
  • Extracts or submits updates
  • In this mode PixieRobot must use the functions of the PixieWeb DLL to operate

By processing instructions or rules written in VBScripts, it extracts data and saves the data in another more structured format. PixieRobot only visits the pages/queries/screens needed, making the extraction processes more efficient and productive, as only targeted data is collected. PixieRobot harnesses the full power of Internet Explorer (IE), and it is difficult to distinguish between its operation and a human user.

PixieRobot can extract pictures and text from Web pages, and process tabular and non-tabular data, and has the ability to automate FORM submits, process buttons/checks/lists, and perform database update processes.

The data gathered from the web or PICK can be stored on a local machine for subsequent processing, for example, storing dynamically in a database, or, sending to another system via email or FTP.

PixieRobot is programmed with VBScript and FREE samples of web scraping scripts are available on request. To access PICK servers the software utility PixieWeb is employed as a smart Telnet connector. Go here to view a live code sample.

Major Benefits and Features of PixieRobot include:

  • Reducing costs by using specialised scripts
  • Speeding up the process by automating the task
  • Eliminating errors and improving accuracy
  • Automated, targeted and unattended processing
  • Navigation to web pages and drilling down
  • Filling-in of forms/screens and submitting them
  • Capture of pictures and data
  • Useful as a host program environment for PixieWeb

When PixieRobot is running interactively, it will display it's console window on the desktop, which is used to control PixieRobot's behavior and monitor progress. As an example the operator may select the script that is to be run, and set other run-time parameters, or view which web page or database file is currently being processed. Alternatively PixieRobot can be set up in the task scheduler to run automatically.

PixieRobot has its own Command set that is used to manipulate the HTML code structure, and when used in conjuntion with VBScript forms a powerful and complete tool-kit.

Data extraction Services

PixieWare Software can write WEB or PICK extract scripts on your behalf. The scripts are yours to keep so you can run as many times as required. You are charged a one-off fee for developing the programs (VBScripts), NOT per transaction extracted.

Other Uses of PixieRobot

PixieRobot can be used as a Console to run any VBscript, whether or not it is data extract oriented, such as reading or updating files or data bases (e.g. MySQL), processing strings of data in general, and many other repetitive and useful administrative tasks. For further information visit our details page.

Note:

If PixieRobot is used to extract (scrape) data from web-sites, we expect all users of PixieRobot to read and comply with those sites existing legal (patent, copyright, trademark) notices and other intellectual property law assertions made by the site owners, in respect of their web-site contents usage. This message does not constitute legal advice, nor is it a substitute for the professional judgment of an attorney should you need assistance.




Discover Our Products and Services

Start I.T.

Web I.T.

Dig I.T.

Contract I.T.

Of I.T.

Grab I.T.

Designed and built by: Designed and built by PixieWare Software

Turn your WEB vision into reality!

Let us
provide
a website quote

Website Build Package
Creation of website (maximum 5 pages). Price: CDN$350 per website, $100 of fee due as an up-front downpayment, and $250 of fee due on project completion. Package does not include any additional external fees related to the project (e.g.):

  • Domain-name registration fee (annual fee)
  • Website hosting fee (monthly or annual fee)
  • Email-accounts processing fee (if relevant, a monthly or annual fee)
  • Any other fee related to domain name re-location or change of name server

Turn your WEB vision into reality!