Network Status News & Information

Update 06.11.20 19.50: The work are now complete. 

Update 06.11.20. 19.00: We are now starting the work. 

----

We are going to update the server prod03.eu.

We don't expect any issues regarding this update, but still we want setup this service window. 

Our plan is to stop the MySQL and Email services on the server in case things doesn't go as planed and we need to do a rollback. 

Email will be queued for delivery once the update is complete by the sender.

Websites using MySQL databases will not be working as the service is shutdown in this service window, static webpages will work as normal.

There may be several reboots on the server in this service window. 

If there is any issue please open a ticket and we will take a look at it as soon as possible. 

The maintenance will require a short interruption of the Internet uplink, we expect an offtime of less than 15 minutes.

12.02.20 - 16.10: We are aware of some unstability on the server. we are investigating. 
--
rebooting server. 
--
16.55: some processess did not respond and the system generated a high load. after the reboot it seems that the system is at a normal state. 
--
17.30: Seems that the issue is still there. we are working on the issue.
--
17.45: found a problem and killed the processes. seems to be running stable for the last 45 minutes. we will continue watching the servers load.

---

19.00: back to normal.

--

13.02.20 - 14.00: Still some problems with the server, we are watching the server. 

--

17.30: Doing another reboot. 

---

14.02.20 - 10.40: Seems that the server is running without any problems. 

rebooting prod03.eu server for system update. 

reboot of server after kernel update. 5 min downtime. 

prod 03 and prod04. 

Upgrading kernel on server. A quick reboot of the server.  

reboot of server after kernel update. 5 min downtime. 

Kernel update on the server. A quick reboot after the update is finished. 

We updated the customer system and migrated the system over to a standalone server. 

we are upgrading the main website. there will be some downtime on the site. 

Will not affect customers webpages.

Knowledgebase articles are not up to date. we are working on getting them up to date. 
If you have any questions or issues please open up a ticket to support. we will try and help you as good as we can!


Open a ticket to support if you got any questions about cPanel and the new control panel.

Upgrading server, must do a reboot. 10 minutes downtime.
--

complete

We will reboot the server for updates. expected downtime for all sites on that server is 10 minutes.
----
Done.

we will reboot the server. downtime will not be any more than 2 minutes.

--
Complete.

We are migrating prod01.eu.livlost.net to prod03.eu.livlost.net. the change will be a new control panel. cPanel. 
When we are migrating you we will inform you on email / SMS. please allow a couple of hours to let the DNS replicate across the network. 

We are doing this manually. Please open up a support ticket if there is any problem. 
Updates will be posted under here: 

----
24.05.17: most of the sites are moved over to the new server. 
Please log in to your controlpanel to check that everything is as it should be. 

Passwords on the email accounts are reset and you will need to use your full email address as username. 

---

31.05.2017 - 21:27
All sites is migrated to the new server. please open a ticket if there is any issues. 
secondary DNS / mail ny.us.livlost.net and prod02.eu.livlost.net has been shut down. 
(downgrading priority from high to low)

--

16.06.2017: legacy servers are shut down. closing issue.

Please open a ticket to support if the server rejects email that is not spam. 

we are tuning the spam filter and want your feedback if we block mail that is not spam.

on 31 May 2017, we will carry out system maintenance on our production systems starting from 7 p.m CEST to account for the implementation of various major & minor updates.

All the systems will be online exept domain info / registering. 

The maintenance should last a few hours.

---

The scheduled system maintenance has been completed.
As of this moment, all services are up & running

we updated the config on the server. when we did that all virtual servers went to the first configured website... 

we updated all the config and the issue should be resolved now.

We will be performing an upgrade of networking software.

During the maintenance window the server will lose networking connectivity for a period of up to five minutes. We expect the actual disruption to be minimal.

server down - prod02.eu.livlost.net

power issue. switches unresponsive. switch replaced.
still working on the issue.


--
hard reboot on the server to get it back up.
-- 
soft reboot to fix a couple of things.

we are upgrading the customer portal. 

livlost.net website will be down in this period.

we have changed the SSL certificate on prod01.eu.livlost.net server. 

you will be asked to approve a new certificate on imap TLS login and smtp login

Mail sent from prod01.eu.livlost.net are ending up in spam folder at some providers. 
we have put in place a temporary solution for delivery, but if the mail it self are considered to be spam this will have no effect! 


Please contact support if there is any problems, from address and to address and the time the mail was sent.

----
Update 05.10.16:
it seems that the issue is somewhat resolved. please open tickets with header info of mails that are not getting delivered.

Update 06.10.16:
seems to be Microsoft Outlook's spam filter. there are no errors in the logs. all mail are relaying as ther should.

The issue is resolved

----

we are unable to provision new virtual servers on our shared hosting platform.

Temprary removed the possibility to pay when ordering shared hosting plans in order to prevent failed creation on a new account
enabled payment, seems to provision for now. we are still looking in to the issue. 

hopfully everything will work again soon.
no services are affected.

The problem will be taken care of ad-hoc. we will maually pick up profisioning that fails.
There may be an issue with adding / deleting users, we will help with that until the problem is resolved. 

Open a support ticket if there is an issue.

we are going to reboot our prod01 server.

Websites will be offline for a couple of minutes.

Enabled on server, DNS zones has been updated. 

Enable IPv6 on server.

Enabled DNS zones too. Closing issue. 

---

Update: IPV6 is up and running on the server. Adding IPV6 DNS Zones

we are adding ipv6 support on the server. there will be downtime within this timeframe.

Update 09.02.2016: Seems to be OK now, closing network issue. 
-----------

Update 06.02.2016: We have disabled the relay and are starting to deliver mail directly from the server. 
please update us with a support ticket if any mail to msn.com, outlook.com hotmail.com are beeing returned. Please add the error message in the ticket. 

-----------
Our prod01.eu.livlost.net are blacklisted at outlook / hotmail.com

we are using our secondary server to relay mail in this period. 

Please open a ticket to support@livlost.net if you have any problems.


warnings on SPF and some other DNS issues is expacted. This is not a solution but a temp workaround.

we are rebooting the secondary servers, ny.us and nl02.eu.

will not affect systems. 

We are in the prosess to migrate fr01.eu server to prod01.eu server.

we will run a backup and restore.

If the migration fails we can do a roll-back as nothing is deleted in the prosess. 

Open a ticket if there is a problem / question. support@livlost.net or your customer portal.

--

Update: Migration is done, please login to your control panel to chack your DNS config.

Final update: everything seems to be in place. please open a ticket if you have any problems.

webserver halts at 06:25 in the morning
one process is not killed and apache wont start. 
the process must manually be killed and restarted. 

we are investigating the issue and will continue to restart the service manually until the problem is resolved.


Update:
logrotate is causing the issue but the real problem are apache 2.4.10. 
we have moved the cron for logrotate to 21.25 at night. and will handle the alerts than.
until a fix in the stable release channel is available we will be handling this manually.
This also happens when a customer updates ther server in webmin / virtualmin (Only the Apache2 / PHP settings).

An alarm goes of after 3 minutes and we will most likely be online again after 10 minutes. 

DNS / FTP / Mail - is not affected.

---

Final update. FR01.EU server is retired. migrated to prod01.eu.

The problem with the main server are too severe, we have to reinstall. This is a good thing tho, a fresh install will resolve the small bugs :)

This will affect all the virtual servers assigned to this server. 
FR01.eu.livlost.net

Email and DNS will still be served by our backup servers (nl02.eu and ny.us) but email accounts / webmail won't be accsessable.

we will first run a complete backup of the server. transfer it to our FTP backup server. 
when this is done we will shutdown mail server and web server including SQL servers. 
a incremental backup will be created and transfered to our FTP backup site. 

we will than begin the reinstall of the server.

For support please open a ticket (NOT email) to our support department. 


Progress and updates in this network issue:
------
GMT + (Oslo)
30.10.15 - 20:00: We begin the first complete backup of all the virtual servers assigned to fr01. 
31.10.15 - 00:00: We are continuing 31.10.15 at 21.00 to avoid peak time. we will start again then. - All systems are working as normal untill we restart the prosess.
31.10.15 - 12:30: Backups are running, it seems that we can push this critical work forward. We are now running at the last stage of a pre-reinstall. a few small things to do and an incremental backup and the reinstall begins.
31.10.15 - 21:00: Starting incremental backup. now. a new update when reinstall are beginning. Mail server and Web server server are shutdown now.
31.10.15 - 23:30: We are starting the reinstall now.
01.11.15 - 00:05: The server has a new install, we're beginning to restore systems.
01.11.15 - 01:35: we are beginning to restore servers from backup.
01.11.15 - 04:40: FR01.eu.livlost.net is back online. a few minor glitches. this will be taken care of in the next couple of days. downgrading severity from critical to medium.
01.11.15 - 21:00: Problems with DNS issues, not transfering DNS zones. we are working on it.


Last update: 01.11.15 - 22:20: Everything seems to be working fine. please create a ticket if you have any issues.

we have problems with apache on this server. it suddenly failes and does not restart the services. 
we are working on this issue. 

this issue is only on the web server part. Mail and DNS are working as before. 

--
GMT +1:
29.10 - 13:35: Rebooting server
29.10 - 16:30: Web server is stable, we are monitoring the server closly. the issue will be updated.
30.10 - 08:30: web server was unavailable again. again, mail / dns is not affected.

30:10 - 19:40: Please se the reinstall issue. the progress will be updated there.

We are upgrading the main site, most likly a downtime on the page for about 10 minutes.

The maintenance  will not have any impackt on services. it will be downtime on this server for about 20 minutes in this window. 

all services will continue as normal. main server (fr01.eu) and secondary #2 (ny.us) will still be online to serve DNS request and receve email. 

please open a ticket if you experience any issues after the maintenance.

The maintenance  will not have any impackt on services. it will be downtime on this server for about 20 minutes in this window. 

all services will continue as normal. main server (fr01.eu) and secondary #2 (nl02.eu) will still be online to serve DNS request and receve email. 

please open a ticket if you experience any issues after the maintenance.

We lost connectivity in 5 minutes. everything is up and running again.

We are refreshing all slave dns servers.
main dns server are still up and running, no config will change there. on the slave servers we will rebuild the database.
This will not have any impact on your services.

update: Working on ns2.livlost.net.
Update, there will take some time, all DNS requests are served by ns.livlost.net and ns1.livlost.net so there will be no impact on services. 
Update: had to reboot ny.us and main server fr01

Update: the issue is solved. resyncing zones across the network. 
update: syncing completed. keeping this open to see if the transfer are syncing as normal ( one times every hour )



Final update: confirmed, DNS is back to normal. all servers are replicating DNS zones as normal. 

We are currently blocked by outlook.com / hotmail.com / live.com due to user reports.

We are looking in to it now. 

A temp end time is set, this will be updated when we know more. 

--

Update: we have contacted microsoft for removal.

--

Update: removal can take 24 - 48 hours to be completed. Setting priority to low.

we lost connectivity to nl02.eu.livlost.net.

this is a backup server, there are two other servers that cover this server.

impact to services are none. 

The issue is resolved. 29 minutes downtime. 

Problem with sending email to domains with hotmail.com / outlook.com..
we have contacted microsoft to have our servers ip removed. 
Updates will follow as soon as we know more.

Update: Microsoft has unblocked our mail server. please open a abuse ticket if there still is a problem.
this can take up ti 48 h to replicate to all of microsofts servers. 

Update: Confirmed resolved.

The loading time on secure.livlost.net are about 20 seconds.

other sites are running as normal. 

we don't know why, but we are investigating the problem.

--

Problem fixed

DNS01.no.eu.livlost.net is a pure DNS management server. 

This will have no impackt on services. DNS on fr01, nl02 and ny.us will still be online and serve DNS queryes.

Expected to be online again 16.00 (GMT +1)

We lost connection to FR01.eu.livlost.net. the server had to be rebooted. 

This issue is now resolved.

We are upgrading the secondary mail and DNS server. it will be down for a couple of minutes in this period.
This will have no impact of services as fr01.eu. and ny.us. server are still up.

we are moving DNS01.no to another node. will be some downtime on DNS01.no

We are shutting down NL01.eu.
all services are moved to nl02.eu and ny.us servers. 

we start by shutting down DNS and mail servers, please open a support ticket if this affects you or your site.

--
The server is completly shut down.

We are moving our Slave DNS/mail server to new server.

both servers are online and will continue to serve DNS requests.

in case of dns resolve/ update issues please open a ticket. 

Seems that the newly setup slave mailserver wont receive mail.
as for now this is not a problem.

we will continue to work on the issue

There was an error on the server witch was resolved. The autosystem did not reboot the server when the error was detected. 

The issue are now resolved 

Power failure on location where the DNS management server are located. 

Does not affect DNS. only management.

Update: Resolved. all services up and running. 

Google are blocking our mail server.
This is a temporary problem and will resolve itself when the traffic returns to normal.

Update:
mail goes through but it takes a little longer than normal.

Update : problem should be resolved now.

Update 23:45: Migration are done. we have to wait for DNS replication.

Open a ticket if there is a problem. 

--------



Migration from de01 to fr01.

we will shutdown the mail server on de01 and the secondary server will take over and store the email.

We hope this upgrade will go smoothly but will keep updating on the progress.

When we start the migration the main site will be moved to amazon with a static page for information. When the migration are done and DNS are updated the site will automaticly become available again. 


You can test to see if the server and DNS are updated by pinging your domain. 
if you see the old ip address the DNS are not updated, Yet.


Old IP : 46.165.245.209
New ip: 62.210.148.168

23.12.13: 00:00 GMT +1. 
Our register came online again. the issue is resolved. 

22.12.13: 17:10 GMT +1. 
Still no updates. seems that our registrar are completly down. we are hoping they are online again very soon.

22.12.13: 14:30 GMT +1.

Seems that there are some problems with domain registering. our registars API are down. 

We are looking in to the problem.

This issue will be updated.

Affects TLD's:

.com
.net
.org
.info
.name
.no
.se
.be
.de
.eu
.es
.uk
.biz
.de.com
.me
.mobi
.pw

We will update the main webpage secure.livlost.net. 
should not be more than 10 minutes downtime on main page. all servers are up and running.

will not affect any customers. 
Client Portal will be unavailable while upgradeing.