Delphi Developers
Components for Internet Component Suite (ICS)

6th July 2022 - OpenSSL 3.0.5 and 1.1.1q Windows binaries released. Note the binaries are now digitally signed by 'Magenta Systems Ltd' instead of 'Open Source Developer, François PIETTE'.

27th May 2022 - ICS V8.69 released

25th May 2022 - Updated PEM Bundle CA Trusted Store Files

There is a new ICS support forum at https://en.delphipraxis.net/forum/37-ics-internet-component-suite/ to replace the old TWSocket mailing that stopped working a few years ago.  

This page contains various components written by Magenta Systems Ltd to extend the François Piette's Internet Component Suite (ICS) version 8 from http://www.overbyte.eu/.  ICS v8 supports Delphi 7, 2006 to 2010, XE to XE8, 10 Seattle, 10.1 Berlin, 10.2 Tokyo, 10.3 Rio,10.4 Sydney and 11.0.  Note that all ICS versions now include SSL free of charge.

Internet Component Suite (ICS) Downloads

All ICS files should normally be accessed from the ICS Download Wiki page but the important files are available here as well.

The latest versions of ICS may be downloaded from the ICS SubVersion server using a subversion client such as TortoiseSVN.  Once your SVN client is installed, you can browse to svn://svn.overbyte.be/ics, svn://svn.magsys.co.uk/ics or https://svn.overbyte.be/svn/ics or https://svn.magsys.co.uk/svn/ics. The SVN login user code is ics and password is ics for read access.

Nightly ICS v8 for Delphi 7-2010, XE-XE8, 10 Seattle, 10,1 Berlin, 10.2 Tokyo, 10.3 Rio, 10.4 Sydney and 11.0
v8 - 6 Jul 2022 (18,123,474 bytes)

Latest ICS V8.69 Release
v8.69 - 28 May 2022 (18,021,895 bytes)

OpenSSL Binaries 3.0 Win32
3.0.5 32-bit - 6 Jul 2022 (2,448,919 bytes)
Major new version of OpenSSL, requires minimum ICS V8.67.  Note the binaries are now digitally signed by 'Magenta Systems Ltd' instead of 'Open Source Developer, François PIETTE' . 

OpenSSL Binaries 1.1.1 Win32
1.1.1q 32-bit - 6 Jul 2022 (1,827,538 bytes)
Only supports Windows Vista/Server 2008, and later, not Windows XP,  requires minimum ICS V8.57. 

OpenSSL Binaries 1.0.2 Win32
1.0.2u 32-bit - 7 Jan 2021 (1,742,205 bytes)
Note ICS V8.65 is the last release to support OpenSSL 1.0.2, whose support ceased in 2019.

Sources of CA Trusted Stores

SSL/TLS trusted root certificate bundles and always changing, annually perhaps for major changes, although Microsoft officially Windows roots every two months. The Common CA Database (CCADB) https://www.ccadb.org/ is a repository of information about Certificate Authorities (CAs), and is used by a number of different root store operators to manage their root stores.

But it's not easy to create root bundles from CCADB and another developer got frustrated with updating roots, and created a Trust Stores Observatory Git repository: https://github.com/nabla-c0d3/trust_stores_observatory which contains over 600 root certificates and lists of which trust store contain which roots by different operating systems. But even this does not contain certificates in a form easily used by OpenSSL, so Magenta Systems Ltd has written a small tool that converts the YAML files from TSO into PEM bundle files, one each for the different operating systems.

New PEM Bundle CA Trusted Store Files

CA PEM Bundle - 25 May 2022 (1,470,800 bytes)

There are six different PEM CA bundle files, built from the Trust Stores Observatory Git repository:

apple.pem - 172 Certificates
google_aosp.pem - 129 Certificates
microsoft_windows.pem - 250 Certificates
mozilla_nss.pem - 136 Certificates
openjdk.pem - 89 Certificates
oracle_java.pem - 93 Certificates

Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store.  Magenta Systems Ltd will periodically update these bundles, as needed. The files are all UTF-8 with a BOM. While the certificates are base64 encoded, the aded comments may include Unicode characters for non-English issuers.

The zip file contains two versions of each bundle, the name above and one ending with -clean.pem which omits all the added textual comments so is smaller and less likely to cause problems with non-English characters.  There are also -titles.txt and -fprints.txt files which are one line per certificate listing the main details, and fingerprint in the latter file.  There are also changes files for the Microsoft Windows bundlle that indicates which certificates were removed or added with each update.

Note the ICS distribution download (see above) contains three CA Trusted Stores, two as PEM bundle files, one in a source unit, and access to the Window Certificate Store directly, see FAQ_SSL/TLS_Certificate_Authority_Root_Stores for more information.

Internet Component Suite (ICS) Release Notes

Changes in ICS V8.69 include:

1 - V8.69 has a lot of minor improvements, more HTTP client and server features, better built-in authentication, and added built in file uploading to the REST client component. Also continuing improving all the HTTP samples so they now hopefully test all the features of the ICS HTTP server, specifically adding POST and PUT file uploading, and a new client login window for interactive authentication with servers. Also added OCSP (Online Certificate Status Protocol) to ICS, which is used to check SSL/TLS certificates are legitimately issued and not revoked.

2 - In the HTTP client, added new methods RequestAsync and RequestSync which start a specified request by parameter, useful to repeat the last request from an event (such as should be trust the certificate and repeat). When using a proxy, make sure Path is not blank which will break proxies, change to / as for non-proxy requests. GetRequestDoneErrorStr now handles TWsocket errors. Fixed NTLM authentication which got broken in V8.61. Added a new THttpAuthType of httpAuthDigest2 for Digest with the modern SHA-256 algorithm instead of MD5 used by httpAuthDigest, only with USE_SSL. Added new property WWWAuthInfos array filled after 401/407 failure by parsing AuthorizationRequest headers for AuthType and Realm that may be presented to select an authentication option, get login and repeat request. Remove # fragment anchor rom the URL unless the new Option httpoAllowAnchor is set.

3 - In the HTTP REST client, added built in file uploading using POST or PUT. The file name is specified in HttpUploadFile using type HttpUploadStrat ofHttpUploadSimple with parameters in the URL or HttpUploadMIME for multipart with parameters in the first MIME part. Note the applications needs to supply parameters like FileName so the server knows what to do with the file. This may be tested against the ICS web server samples. Upon request completion, ReasonPhrase now also has status, so OK becomes 200 OK, etc. Previously the SslRevocation property was only effective when checking the windows certificate store, now it also works with bundle files using the new TOcspHttp component and OCSP stapling if available.

4 - In the HTTP server component, now converting FLastModified to UTC/GMT time for the response header. Added new authentication type atDigestSha2 with a SHA256 hash instead of MD5 with atDigest only supported with USE_SSL. Note: Mozilla Firefox supports Digest SHA-256, Chrome and Edge do not. Added a new client connection OnHttpRespHdr event to allow response headers to be logged, previously only request headers could be logged. Cleaned up AnswerStream functions so separate lines don't go into send buffer. Added OcspSrvStapling property which should be set to enables OCSP checks and stapling only with AUTO_X509_CERTS define since it adds extra HTTP client code. A revoked certificate will be auto ordered.

5 - In TWSocket, added new method SendTB(const Data: TBytes; Len: Integer=-1) where Len is optional, also similar SendToTB and SendToTB6 with TBytes, as an alternative to casting such types to use Send(). Restored the TlsExtension_cb callback for client debugging since ClientHelloCallback only works with servers.

6 - In TSslWSocketServer added OCSP (Online Certificate Status Protocol) support with IcsHosts using the TOcspHttp component to confirm server SSL/TLS certificates are legitimate and not revoked for security reasons. The certificate OCSP response is also stapled to the initial SSL/TLS HELO handshake and sent to the client to avoid it needing to lookup OCSP using HTTP itself. OCSP responses are cached and saved to a file for reloading later, but are refreshed every time the certificate is validated, at least once a day. The new server property OcspSrvStapling enables OCSP checks and stapling only with AUTO_X509_CERTS define since it adds extra HTTP client code. A revoked certificate will be auto ordered. OCSP checking is done in LoadOneCert and the stapled response sent in TriggerSslServerName when checking SNI. The same OCSP support is available in all ICS servers that use IcsHosts, including FTP, HTTP and TIcsIpStrmLog.

7 - All three main HTTP SSL client sample applications FrameBrowserIcs, OverbyteIcsHttpsTst and OverbyteIcsHttpRestTst now support interactive authentication with a new Login window that displays the different methods the server will accept (from the new property WWWAuthInfos array) allowing one to be selected from Basic, Digest MD5, Digest SHA256 and NTLM logins. This window is displayed after an 401 error and the request then repeated.

8 - The OverbyteIcsHttpsTst SSL sample now has all the missing features from the non-SSL samples, Content Encoding Gzip tick box to support compression, persistent cookie support, POST/PUT support to either send simple data or upload files in various ways (from OverbyteIcsHttpPost1 sample), may be tested against the ICS web server samples.

9 - The OverbyteIcsDDWebService SSL web server sample now builds on unicode compilers. Added authentication for POST requests and new 'Password protected page (POST)' button on the demo menu to test authentication using POST. Fixed web logging to log correct multiple listener. Builds on unicode compilers. Added Digest SHA-256 authentication page DemoDigest2Auth.html and DemoDigestsAll.html that does both digests. DemoAuthAll.html no longer does NTLM, use the separate page. Now displays server response headers if box ticked, only displayed request headers before. File Upload Form and Email Form pages now work without exceptions. The OverbyteIcsSslMultiWebServ SSL web sample has similar new authentication features and logging fixes.

10 - Updated OpenSSL to 3.0.3 and ZLIB (HTTP compression) to 1.2.12, sorry for long delay in updating zlib, now including some important bug fixes, although never saw any issues with the minimal use ICS makes of it. Updated the various ICS CA Trusted Stores.

11 - The OverbyteIcsPemtool SSL sample now does OCSP checks when examining certificate files. There is also a new Test Host Certificates tab that tests SSL/TLS handshake and certificates (using TIcsIpStrmLog) for a list of host names and ports, building a list of host and intermediate certificates that may be further checked and saved as files or added to an intermediate bundle. This new tester should prove useful for debugging sites that return SSL or certificate errors.

More detailed release notes are at ICS 8.69 Release Notes

Changes in ICS V8.68 include:

1 - V8.68 is a minor release, mainly HTTP client and server improvements including new request and response headers to assist browser caching and conditional requests, improved error reporting to help diagnose failed HTTP requests, and improvements in the HTTP REST component allowing it to save files including resuming failed downloads and download files of any size. There are minor fixes in various components, and updates to installing on MacOS and C++ Builder. There is also a new File Clean-Up demo ideal for deleting the old log files that many applications (including some ICS samples) leave on systems.

2 - The HTTP client TSslHttpCli now keeps the Etag response header as ResponseEtag, allowing applications to save it with the page content, and when requesting a refresh to add the request method ReqIfNoneMatch or ReqIfMatch to avoid downloading the page again. Made RequestDoneError available as property so it can be accessed after Sync requests, and added RequestDoneErrorStr property that returns a literal error. Improved RequestDoneError to give more information than a simple abort, added httperrOutOfMemory and httperrBgException which happen while receiving and processing data in the OnDocData event and httperrSslHandShake. Improved AbortComponent so ReasonPhrase now reports the exception that caused it, such as out of memory which previously needed a BgException event handler. For range downloads, added the ContentIfRange request header which can send an RFC1123 date or Etag so a partial download only happens if the file is unchanged.

3 - Previously, the HTTPS REST client TSslHttpRest always downloaded content to a TMemoryStream with content size being limited to MaxBodySize (default 100 MByte), and generally restricted by memory to less than 250 MByte. To remove this limitation, added the HttpMemStrategy property with THttpMemStrategy on how to handle downloads: HttpStratMem only TMemoryStream; HttpStratTemp uses a work file in the system temporary directory for sizes larger than MaxBodySize; HttpStratFile always writes a named file HttpDownFileName (with .part extension during download); HttpStratResume is similar to HttpStratFile but supports resume of failed partial downloads (with .http extension for resume information). Property ResumeMinSize defines the minimum sized partial file that should be resumed, rather than start again (default 64K). Note MaxBodySize remains the maximum size for ResponseRaw (unicode string), JSON and XML parsing. Note this is a breaking change since the ResponseStream property is now TStream, so to use the SaveToFile method you will need to be cast as TMemoryStream. Or change to use HttpStratFile method with HttpDownFileName. Also note, ResponseStream remains open after request completes and may occupy a lot of memory or leave file open (read only), also ResponseRaw string, so use ClearResp method when no longer needed. The component now logs a better error if the response does not contain Json. Added ShowProgress property that causes download information to be sent to the OnHttpRestProg event using LogOption=loProgress, showing progress in KBytes of KBytes updated every ProgIntSecs seconds, default two, expected to be displayed as a caption. The OverbyteIcsHttpRestTst.dpr sample has fields to select a Download File Name when saving a file and for Memory Strategy, download progress is also displayed.

4 - The HTTP server TSslHttpServer now supports the If-Range, If-Match and If-None-Match request headers for conditional pages using Etag or last modified date, and sends 304 not modified for matches, to help with caching. Added an ETag header to responses in AnswerStream and AnswerPage when we can create one from a file modification date and size (base64 CRC32), or if the EntityTag property is specified in the client onGetDocument event before using hgSendDoc or hgSendStream, perhaps a CRC32 of the entire content from a cache. The 304 not modified response now includes more recommended headers. Also add a Date: header to AnswerStream and AnswerPage responses to help with caching. Note, if applications already add Date: or Etag: to CustomHeaders or to an AnswerStream should remove them to avoid duplication.

5 - The Multi HTTPS client  TIcsHttpMulti has been updated to a fix a problem that meant large files failed download with only an abort error if too large for TMemoryStream, now downloaded to TFileStream with .part extension and renamed up successful completion. If a partial file downloads, don't delete it if KeepPartDown=True. Added logging of RequestDoneError on failure and BgException. Correctly download files using chunked coding without known size, note these show with size -1 since not known until complete. These fixes were triggered by failed HTTP 300 MByte downloads from a new HD CCTV camera that said abort with no mention of out of memory errors, thus a lot of debugging and ICS changes to improve matters.

6 - The previous ICS release added support for the OpenSSL 3.0 release. Now it's been available for three months it has been added to the main distribution, the samples SslInternet directory now has both OpenSSL 1.1.1m and 3.0.1, ICS will try and load OpenSSL 3.0 first, then 1.1.1 if not found, unless the global variable GSSLEAY_DLL_IgnoreNew is set true before OpenSSL is loaded. Likewise GSSLEAY_DLL_IgnoreOld may be set true to ignore 1.1.1 and fail unless 3.0 is available. ICS now supports YuOpenSSL 3.0 and 1.1.1 versions as commercial DCUs allowing applications to be used with OpenSSL without needing separate DLLs, except it still loads providers like legacy.dll as DLLs, not a DCU.

7 - In TIcsWndControl, BgException now passes the exception to the AbortComponent virtual handler and ExceptAbortProc method so higher level components can report errors better, such as out of memory without needing an BgException handler. Note requires similar change to all units that override AbortComponent or use ExceptAbortProc.

8 - The TX509Base component now supports OpenSSL 3.0 for Load/SaveP7BFile. ValidateCertChain no longer reports an error for the expired 'DST Root CA X3' CA root, since some platforms accept expired roots. Improved OpenSSL error handling to say 'No error returned' instead of error:00000000:lib(0):func(0):reason(0). TWSockey now clears LastError in Listen and Connect methods to avoid false errors later. Improved source description for some fatal background errors.

9 - In TSslX509Certs, added property KeepOldCA set true to keep the old Let's Encrypt intermediate for the expired DST Root CA X3 root in bundles to support old Android releases. Keeping it may prevent some clients verifying the chain and SslLabs testing gives a chain warning.

10 - In TIcsFileCopy, the DeleteFiles method now supports a new Zipped property so files are zipped with .zip extension before being deleted, useful for cleaning up old logs to save space (requires VCLZip). The OverbyteIcsXferTst,dpr sample has a new File Clean-Up tab to test the multiple DeleteFiles method, it allow files older than x days or a  specific date range to be archived/zipped (to save space) or deleted, ideal for deleting the old log files that many applications (including some ICS samples) leave on systems.

11 - In the FTP client TSslFtpClient, using the PORT command to set Active mode now prevents other FTP clients sharing the same port number and address, usually only a problem with a small port pool (which is not recommended). For an Abort, LastResponse now reports the exception that caused it, such as out of memory which previously needed a BgException event handler.

12 - In the Multi FTP client TIcsFtpMulti, improved TLS session caching by ignoring the connection port so the data sessions can use the control session.

13 - Updated the trusted root certificate bundle files, lots of changes from Microsoft since June, Google is now issuing it's own certificates. Updated the build-in sslRootCACertsBundle, few gone, now total 59 certificates commonly used.

14 - ICS should now build with C++ for RAD Studio 10.4 and 11.0, fixed the 11.0 packages and various Windows API related units, including for Win64. The Platform FMX samples now build with RAD Studio 11.0 where some FMX properties have changed, not sure if they'll still build with older versions. Building samples no longer needs the source search path to have \Include, this has been added to all the sample units that use OverbyteIcsDefs.inc.

More detailed release notes are at ICS 8.68 Release Notes

Major Changes in ICS V8.67 include::

1 - Added support and packages for RAD Studio 11.0. Updated SSL/TLS root certificate bundles, old certificates gone, new ones added, nothing major.

2 - Added support for OpenSSL 3.0 which is a major new release, primarily a lot of internal changes to ease long term support. There is an optional FIPS module with 3.0 but not available here since our DLLs are not built to the standards required for certification. The old engines for special extensions are replaced by new more versatile providers of which the FIPS module is one, a provider legacy.dll has obsolete ciphers and hash digests, including MD2, MD4, Blowfish, DES, IDEA, RC2, RC4, SEED, that most applications no longer need and which needs to loaded by the application by setting global variable GSSLEAY_LOAD_LEGACY to true before loading OpenSSL.

3 - OpenSSL 3.0 does not offer any specific new features of benefit to ICS at present, although HTTP/3 support is planned for 3.1 or later, so the main ICS distribution retains OpenSSL 1.1.1i which is fully supported until September 2023. OpenSSL 3.0 may be downloaded from the download page.

4 - The main implication for ICS with OpenSSL 3.0 is for SSL/TLS certificate private keys saved with password protection, which is required for PKCS12 certificates for importing into the Windows certificate store. The new PKCS12 default password encryption AES256 is not recognised until Windows Server 2016 v1709 and Windows 10 v1709, so Server 2012, Windows 10 RTM and earlier won't load AES passworded keys, only 3DES, for which the legacy.dll must be loaded.

5 - There are two new classes to write and read SSL/TLS certificates to and from the Windows Certificate Store, including private keys. This is primarily so Let's Encrypt certificates can be installed automatically for use with the IIS web server.

6 - Various improvements for the OverbyteIcsPemTool sample. It includes new buttons to list the contents of Windows certificate and private key stores and allow old items to be deleted. This may be useful for cleaning up old certificates and private keys from the Windows stores.

7 - For the TX509Certs component, the default cipher for encrypting PFX/P12 files is now PrivKeyEncAES256 with 3.0 unless the legacy DLL is loaded when still PrivKeyEncTripleDES so older versions of Windows can load them. Changed extraction of download PEM bundle so that main certificate does not need to be first in file, log them all, and ignore any self signed root certificates.

8 - Fixed two problems in the FTP client, support option ftpFixPasvLanIP for PUT/APPE uploads as well as downloads, and support IPv6 for PUT/APPE uploads as well as downloads.

9 - In the Application Web Server TSslHttpAppSrv, added an optional LastModified parameter to the AnswerStream, AnswerPage, and AnswerString methods to avoid adding a custom header line with the date. Added NO_CACHE_EX and NO_STORE_EX literals. Added PUT and DELETE verb handlers, similar to GET and POST.

10 - For the HTTP client TSslHttpCli, fixed a relocation problem where the Location: header included a path with a space, encode the space. Fixed another relocation problem where HEAD sometimes stalled. Remove # fragment or anchor from URL in relocation, only used by browsers and not by servers.

11 - Added a new SSL sample, OverbyteIcsDDWebService.dpr which is very similar to OverbyteIcsSslMultiWebServ.dpr, but designed as a Windows service, although
it will also run as a GUI for debugging. It requires DDService service framework to be installed from ddservice.asp. It also includes a REST server with simple lookup responses from a SQL database, which optionally requires DISQLite3 5.36.5 or later to be installed from http://www.yunqa.de. Note this sample in not in the project groups due to
these pre-requisites.

12 - Moved TRestParams from the OverbyteIcsSslHttpRest unit to OverbyteIcsUrl to ease circular references. Added a new method AddItemNULL to add a null, in Json this will be unquoted. Added a new TRestParamsSrv component which provides methods for creating REST server Json responses from a SQL database resultset, one or more rows, also error responses. Note this is only compiled if DATABASE is defined in OverbyteIcsDefs.inc to avoid bringing in database units that are not available on all Delphi editions. There is a REST server sample OverbyteIcsDDWebService.dpr that illustrates SQL lookups.

13 - In the proxy component TIcsHttpProxy, don't send an HTTP request header until after HTTP body has been processed in case the body length changes. HTTP Forward Proxy using HTTP works again, broken in V8.65. Using HTTP Forward Proxy, convert absolute URL to path only since some servers can not process an absolute URL and sulk.

14 - In the Jose unit, rewrote the functions converting private keys to and from Json Web Keys with new OpenSSL 3.0 provider functions. Use AnsiStrings and functions when dealing with binary data to avoid possible issues with string conversions and nulls. Json now created with TRestParams.

15 - Added two new sample project groups, OtherDemos64 and SslDemos64 which include Win64 versions of all the main active samples with 64 added to the project name, so they can be regularly built alongside the Win32 versions without changing platforms and overwriting executables.

More detailed release notes are at ICS 8.67 Release Notes

Major Changes in ICS V8.66 include:

1 - Added a new TIcsInetAlive component to check for IPv4 and/or IPv6 internet connectivity, using Ping and/or HTTP, defaulting to www.msftconnecttest.com run by Microsoft for Windows 10 alive checking. The online and offline check intervals may be set, and event fires when online state changes. Sample OverbyteIcsHttpRestTst has a demo for new component.

 2 - OpenSSL 1.0.2 and 1.1.0 ceased security fix support over 12 months ago so ICS now only supports 1.1.1, with 3.0 support due in the next few months.This removes a lot of legacy code and functions, and several old ICS functions needed for backward compatibility.

3 - Undertook a major clean-up of OpenSSL functions that may require end user application changes if low level OpenSSL functions have been used, hopefully very rarely. All OpenSSL functions have been renamed to their original names removing ICS f_ prefix for commonality with other Delphi applications.

4 - Bring server SSL/TLS security levels up to latest Mozilla recommendations. In 2021 Mozilla now recommends TLSv1.3 as modern ciphers and TLSv1.2/1.3 as Intermediate supporting all browsers from last five years, so IcsHosts now use sslCiphersMozillaSrvTLS12 as Intermediate level, also Mozilla recommends no cipher server preference so changed that.

5 - Added support for YuOpenSSL which provides OpenSSL in a pre-built DCU statically linked into applications, rather than using external OpenSSL DLLs. This make application distribution more reliable since it can no fail by users deleting the DLLs or copying incompatible versions into the directory. YuOpenSSL is a commercial product from https://www.yunqa.de/ and is supplied as separate compiled DCUs for Delphi 5 to 10.4.

6 - Added a better way of configuring Socks proxy and HTTP Tunnel proxy settings in TWSocket, similarly to that added for THttpCli in V8.62. The new ProxyURL property sets SOCKS or HTTP Tunnel proxy settings using a single URL, ie proto://[user:password@]host:port where proto is socks5 or http.

7 - TIcsRestEmail has new OAAuthType property and event for OAuth2 browser URL perhaps via email for servers. TIcsRestEmail now sets OAuth2 errors, and  clears old tokens.

8 - TIcsHttpMulti has a new ParseLevels property that follow links on a parsed web page to lower level pages, which are also parsed, looking for files to download with a specific file extension, ie .avi.

9 - Updated to OpenSSL 1.1.1k with two high security fixes.

More detailed release notes are at ICS 8.66 Release Notes

Detailed V8.65 Release Notes
Detailed V8.64 Release Notes
Detailed V8.63 Release Notes
Detailed V8.62 Release Notes
Detailed V8.61 Release Notes
Detailed V8.60 Release Notes
Detailed V8.58 Release Notes
Detailed V8.50 Release Notes

Now part of ICS V8.60 and later, as TIcsMailQueue

Magenta Systems Mail Queue Component
Version 2.5 - 26 Nov 2018 (2,414,770 bytes)

Magenta Systems Mail Queue Component has two main benefits over a simple TSslSmtpCli component: it supports extended retries over many hours or days, and supports multiple SMTP relay servers or looks-up MX servers, while alleviating the need for the application to handle retries. The component also allows HTML mail to be sent using SSL, something THtmlSmtpCli does not currently support. Mail is queued to disk, so retries will continue if the application is restarted.

TMagMailQueue is designed to prepare, queue and send email. Preparing the email is done using the ICS THtmlSmtpCli component so it may be plain text or HTML email with one or more file attachments.

Once the mail properties in QuHtmlSmtp have been specified, it is queued using the QueueMail method which saves it to an EML spool file.

The component runs a thread which checks the queue for new EML spool files, and attempts to forward them to one or more SMTP Mail Servers using TSslSmtpCli, optionally with SSL. If mail delivery succeeds, the spool file may be deleted or moved to an archive folder. If mail delivery fails, the spool file remains in the queue and further attempts are made separated by the times in minutes listed in the RetryList list. If all delivery attempts fail, the spool file may be deleted or moved to a badmail folder.

Note that some email servers support grey listing and reject the first email attempt from a new sender but allow a retry 10 or 15 minutes later, something that is very effective in blocking spam emails (since they don't usually retry).

If multiple mail servers are specified, delivery is attempted once using each server, for each retry attempt. Each mail server is specified as TMailServer and there is no limit to the total.

Each time the queue is updated or a delivery attempt made, the queue is saved to file in the control folder, so the component may be stopped and restarted with failed attempts continuing.

The EML spool files are compatible with those created by many Microsoft email applications such as CDO, and the AddtoQueue method can also be used to queue existing EML files with the queue details specified in MailQuItem.

Note, this component is intended for sending low volume email from individual Delphi applications, with more flexibility than a simple TSslSmtpCli component. For use as a heavy duty SMTP server, queue processing could be improved to avoid moving records around as much or saving them to disk as often, and mail bodies could be read as required from disk instead of being read entirely to memory first.  A mail pickup folder could be added which is scanned for new EML files.

Files and Folders Used

The TMagMailQueue component heavily uses disk files, in different sub-directories within the mail root directory specified in property MailQuDir, these are:

• control - contains MailQuItems.Ctl a single row file with the next message item number, and MailQuItems.Hdr which is a CSV file containing one row for each mail item still in the queue.
• spool - contains any queued email files, named in the format item00000001.eml with the number increasing, taken from MailQuItems.Ctl
• archive - if ArchiveSent property is true, once an email has been successfully sent it is moved into the archive directory
• badmail - if DeleteFailed property is false, once an email has exceeded all the retry attempts it is moved into the badmail directory, from where it may be manually requeued if necessary

If logging of sent email is specified, the default file name FileQuSent property is MailQuSent-yyyymmdd.log inb CSV format similar to MailQuItems.Hdr.

A demo application mailqudemo.exe illustrates simple email queuing.  The zip contains the EXE demo and required SSL files.

Release Notes

18th January 2011 - 1.0 - first public release. Not yet tested with Delphi 2009 or later.

2nd March 2011 - 1.2 - automatically create mailqueue directory in demo application, removed missing uses statement. Support queuing mail with OwnHeaders bypassing htmlmail. Log event definition changed.

11th August 2011 - 1.2 - updates subroutines for Win64 support, removed one unneeded unit from uses.

5th Oct 2011 - 1.3 - Debug logging works properly Don't retry emails that fail too large for server (error 552)

11th Sept 2012 - 1.4 - ICS V8, IPv6

23rd March 2013 - 1.5 - Added Mail Server SocketFamily and LocalAddr6 for IPv6

10th Dec 2014 - 1.6 - Better SSL handshake reporting

27th Oct 2015 - 2.0 - requires ICS V8.19 October 2015 or later.
Check and report SSL certificates using PEM file or Windows Cert Store
Allow three SMTP servers to be specified for each email in queue
Lookup DNS MX records and send to those SMTP servers
Queue keeps last response or error in queue
Mail completed log (same CSV format as queue)
Queue changed event to tell client something is happening
QueueMail method now returns item number (not boolean)
New UnQueueMail method to remove item number from queue
Demo save settings in INI file
Demo new View Mail Queue window to see what's waiting
Added SMTP Send Method, relay, specific or lookup MX mail servers
Added HELO Sending Host Name may be needed if using MX mail servers

Warning - if using MX DNS servers and multiple recipients, need to queue mail multiple times !!!! This will be fixed real soon.

7th July 2016 - 2.1 - requires ICS V8.30 July 2016 or later.
Support SSL enhancements in ICS for OpenSSL 1.1.0
Don't change SSL directory, let application control it
Use default SSL root bundle if none specified

1st December 2016 - 2.2 - requires ICS V8.39 November 2016 or later.
Better error handling.
Use OpenSSL host checking.
Fixed bug that meant failed email was not deleted from queue.
Don't queue email without recipients.
Use timer to update windows to avoid problems with mass email performance.

6th March 2017 - 2.3 - requires ICS V8.43 March 2017 or later.
Simplified SSL certificate reporting.

11 Mar 2017 - 2-4 - Added WaitSend to wait until everything sent.

22th Jun 2018 - 2.5 - requires ICS V8.55 20 June 2018 or later.
Added RetryWithoutSsl which retries an SSL failure without SSL.
Added SslCliSecurity to set client security level.
Using IcsWndControl for threaded message handling.
SendSmtpClient now created new for each attempt in case of prior faillure causing terminal corruption.
If SSL certificate verify fails, next attempt is another server.
Supports TLSv1.3 with OpenSSL 1.1.1.

26th November 2018 - 2.5 - tested with ICS 8.58
Added final OpenSSL 1.1.1a DLLs, recompiled.

Now part of ICS V8.60 and later. 

Now part of ICS V8.60 and later, as TIcsIpStrmLog

Magenta Systems IP Log Streaming Component
Version 2.8 - 14 Dec 2018 (4,156,649 bytes)

TMagIpLog is designed for IP stream logging, using TCP Client, TCP Server, UDP Client or UDP Server protocols, sending simple text lines across a network so they may be displayed or written to disk remotely. The component allows two way communication with TCP and UDP, so may also be used for simple protocols such as communication between two applications. The component supports multiple client sockets so may be used to send data to two or more different remote servers at the same time.

 For TCP and UDP clients, the component will optionally ping the remote computer first before opening an IP connection to allow faster failure retries and some confirmation that UDP may work. TCP client provides repeated connection retry attempts, including re-establishing a lost connection. UDP client will optionally keep pinging the remote during a connection to ensure it's still there. UDP server sends data to the IP address and port from which it last received data. TCP server supports multiple remote clients connecting. Received data is parsed for various line endings optionally removing control characters and triggering an event for a received line. The only other two events are optional, one for state changed when starting and stopping, the second offering progress information and errors. 

The component supports both IPv4 and IPv6, host name lookup for TCP and UDP Client, and SSL connections for TCP Client and TCP Server, including remote server certificate checking using either a local PEM bundle root file or the Windows Certificate Store.

A demo application testiplog.exe illustrates use of TMagIpLog as a TCP or UDP client or server, and both in the same program sending data locally. The same component may be used in a client or server application, to send or receive.

The Magenta Systems ComCap application may also be used to capture IP streams to files or a database.

Using TMagIpLog:

1 - Drop the component onto a form (or create it in code, see testiplog.exe).

2 - Specify LogProtocol as one of logprotUdpClient, logprotUdpServer, logprotTcpServer, logprotTcpClient.

3 - For client protocols, specify RemoteHost (name or IP address) and RemoteIpPort, CheckPing true if ping to be used, RetryAttempts to non-zero if continual retries not needed, RetryWaitSecs for delay between retries .

4 - For server protocols, LocalIpAddress is 0.0.0.0 to listen on all local addresses, LocalIpPort must be non-zero.

5 - For sending data, AddCRLF to false if line already have terminating characters, UdpNoCRLF to false if UDP should send CRLF.

6 - For receiving data, LineEndType to one of lineendCR, lineendLF, lineendCustom (set in hex in CustomLineEnd) or lineendPacket (for UDP), then MaxLineLen if a line should be returned before lineend is found, normally non-ASCII characters are removed, set StripControls to false if they should be replaced by spaces, RawData to true if CR, LF, FF and control characters should not be removed.

7 - Assign onLogRecvEvent if data is to be received, onLogChangeEvent if tracking of start and stop is needed, onLogProgEvent if progress information is needed for logging.

8 - Call StartLogging. The LogChangeEvent and LogProgEvent will trigger when LogState changes to logstateOK when data may be sent.

9 - To send a line, if function GetAnyStateOK is true, call SendLogLine. MaxSendBuffer specifies the amount of data that can be buffered otherwise SendLogLine will fail.

10 - Received data will trigger LogRecvEvent once per line.

11 - Call StopLogging to stop. Buffered data may continue to be sent after close, keep calling CheckStopped until true when it's really finished and component may be destroyed.

12 - To send an unlimited size stream, create a stream in the application with TBufferedFileStream or TFileStream, and pass it to SendStream.  LogState changes to logstateOKStream while it's being sent, then back to logstateOK as it finishes, the application should then free the stream.

13 - There is no specific handling for receiving a stream, textual data will be handled according to the normal line end properties, and can be saved to another stream in LogRecvEvent. Binary data is more problematic, set RawData to true and MaxLineLen to get a buffer load at a time, but the last buffer load will need to be extracted with GetPartialLine using a timeout, this is called automatically when the connection is closed.

14 - To send to multiple clients, set MaxSockets to the number needed, then use the function SetRemotes to specify the remote host and port for each socket number, base 0. The events all return Socnr to indicate which socket. MaxSockets also specifies how many remote clients can connect to TCP Server, but note that Socnr is dynamic and changes as remote clients come and go.

15 - To support SSL on TCP/IP client or server, drop an TSslContext component on the form, assign it to the LogSslContext property and set the ForceSsl property to true. For better performance, set LogSslSessCache to a TSslAvlSessionCache component.

16 - For SSL TCP Server, the SslContext component must have the SslCertFile and SslPrivKeyFile properties set to the file names of an SSL certificate and Private Key PEM files respectively, and SslCipherList set to sCipherMozillaSrvBack for strong but backward compatiblle cipher support. The component includes sample self signed certificate and password files iplog-cert.pem and iplog-prvkey.pem, and you can create your own with the ICS SSL sample application Pemtool, or buy commercial PEM certificates.

17 - For SSL TCP Client, the SslContext component must have the SslCAFile property set to the file name of a PEM root certification authority file containing trusted root certificates. Such a file is supplied with the component RootCaCertsBundle.pem containing various root certificates covering most major registries. SslContext SslCipherList can be left as the default to allow connection to any server. The LogSslVerMethod property can be logSslVerNone to skip certificate verification, logSslVerBundle to check using the CA bundle file or logSslVerWinStore to check using the Windows certificate store (a little slower, bur maybe more certificates). To check if certificates have been revoked set LogSslRevocation to true, beware this needs public internet access and can be very slow or fail. LogSslReportChain set to true reports certificate details checked.

Release Notes

18th August 2007 - 1.1 - using OverbyteIcsFtpSrvT instead of OverbyteIcsLibrary, UDP receive packets may be from multiple hosts, always keep IP.

5th August 2008 - 1.2 - made compatible with ICS V7 and Delphi 2009. Note only supports ANSI with Delphi 2009.

20th August 2009 - 1.3 - fixed problem with MaxSockets being reported as closed in the event when only one was open, tested with Delphi 2010.

9th August 2010 - 1.4 - removed cast warnings with Delphi 2009 and later

22nd Sept 2011 - 1.5 - added SndBufSize and RcvBufSize to increase buffer sizes and speed

11th Sept 2012 - 1.6 - better error for too many clients with server added CurSockets property for current number of server sockets

7th July 2014 - 2.0 - now only ICS 8 and later, using new ICS ping. 
Added IPv6 and SSL support, including server certificate checking.
Added host name support for UDP and TCP client with DNS lookup.
Added LogProtocols suffixed 6 for IPv6.
Cleaned up some progress messages, identify error progress events.
Removed line length limit of 1024 that was not checked.
Added send a stream of unlimited length.
Get buffered partial received line during close.
Default line end is LF instead of CR so UNIX files are processed.

13th July 2015 - 2.2 - requires ICS V8.18 June 2015 or later.
Added better SSL handshake error reporting.
Added lineendCRLF, only support FF as lineend if using CR.
Added Debug Info button for ICS info level logging.
Added SSL Server DH Params, set ECDHCurves, both for ECDH ciphers.
Note OpenSSL no longer support dhparam512, minimum is 768 bits.

23rd Oct 2015 - 2.3 - requires ICS V8.19 October 2015 or later.
Better SSL client and server certificate reporting.

8th July 2016 - 2.4 - requires ICS V8.30 July 2016 or later.
Fixed certificate reporting typo.
Removed TBufferedFileStream, not needed.
Added SrvTimeoutSecs to close idle server sessions, note needs ICS V8.30  or later to fix a SSL bug that stopped SrvTimeoutSecs working.
Added Socket property to get current socket, mainly for statistics
Report session length and data xmit/recv before closing

23rd Nov 2016 - 2.5 - requires ICS V8.39 November 2016 or later.
Added GetSendWaiting to check how many bytes of send data not yet sent.
Increased default MaxSendBuffer size to 64K.
Added property TotRecvData total data received since connection, or when method ResetRecvData was called.
Added property MaxRecvData which causes onLogRecvEvent to be called when that length has been received. May be used for fixed length binary packets or where received data contains a content length such as a HTTP response header followed by binary data.
Server takes exclusive access of addr/port.
Fixed bug with multiple clients not using correct port.
Added SSL Server Name Indication support.
Check multiple client SSL host names correctly.
Removed USE_SSL so SSL is always supported.
Removed TX509Ex now using TX509Base.
Using OpenSSL certificate verification host checking.
Server now supports LogSslReportChain to report server certificates, checks expired and reports chain.

7th March 2017 - 2.6 - requires ICS V8.43 March 2017 or later.
set IcsLogger for context so it logs more stuff.
Simplified reporting SSL certs in client handshake.
Improved validation of server certificates.
Use threaded DNS lookup.

22nd June 2018 - 2.7 - requires ICS V8.55 20 June 2018 or later.
Support TLSv1.3, no real changes.
Don't start SSL handshake twice.
Cleaned up SSL error handling.
Added SslCliSecurity to set client security.

14th December 2018 - 2.8 - tested with ICS 8.58
Added final OpenSSL 1.1.1a DLLs, recompiled.
Removed madexcept.

Pending major changes to use IcsHosts in 3.0.

Now part of ICS V8.60 and later.  Uses IcsHosts.

Now part of ICS V8.60 and later, as TIcsWhoisCli with a new sample application

Whois Component and Demo
Version 1.0 - 2 Nov 2005 (254,079 bytes)

A Whois component and demonstration application.  Whois is a protocol to interpret a remote server for information about a domain name or an IP address, and return textual information about 'owner' of the name or address.  The demo application interprets the result and will perform a secondary query to another Whois server if necessary.

Now part of ICS V8.60, as TIcsTimeClient and TIcsTimeServer with a new sample application

SNTP Time Server and Client Components
Version 1.0 - 9 Mar 2006 (11,405 bytes)

TTimeServ is an updated version of Nathan Anderson's time server component adding SNTP support.  TWSTimeClient is an updated version of Chris Barber's time client component adding SNTP support and functions to change the PC UTC time.  SNTP provides time correction with fractional seconds, unlike the earlier Time protocol that is round seconds only.

Magenta Systems Ltd, 9 Vincent Road, Croydon CR0 6ED, United Kingdom
Phone 020 8656 3636, International Phone +44 20 8656 3636
https://www.magsys.co.uk/
Copyright © 2022 Magenta Systems Ltd, England. All Rights Reserved.