After upgrade from 14.04 to 16.04 onion site doesn't work

smbd · 2017-06-06 15:44:51 UTC

The log contains:

2017-06-05 23:09:24+0200 [-] &#39;[!] exception mail suppressed for exception (12d7d37fc7d510d04d912de85ee855b0fff9814ae79f2450f2e9d4706d228b55) [reason: threshold exceeded]&#39;
2017-06-05 23:09:28+0200 [-] [http] 200 POST &#47;receiptauth (127.0.0.1) 152.73ms
2017-06-05 23:09:28+0200 [-] Unhandled error in Deferred:
2017-06-05 23:09:28+0200 [HTTPConnection,39,127.0.0.1] &#39;[!] Unhandled exception raised:&#39;
2017-06-05 23:09:28+0200 [HTTPConnection,39,127.0.0.1] &#39;[!] twisted.internet.error.ConnectionRefusedError Connection was refused by other side&#92;&#92;n&#92;&#92;nConnectionRefusedError: Conne
ction was refused by other side: 111: Connection refused.&#92;&#92;n&#39;
2017-06-05 23:09:28+0200 [HTTPConnection,39,127.0.0.1] &#39;[!] exception mail suppressed for exception (f9527d64c578191cd812f15091ba8f61b2b3cf8321c0ae8f0b08687a861f20a4) [reason: threshold exceede
d]&#39;
2017-06-05 23:09:28+0200 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

2017-06-05 23:09:28+0200 [-] Unhandled error in Deferred:
2017-06-05 23:09:28+0200 [HTTPConnection,39,127.0.0.1] &#39;[!] Unhandled exception raised:&#39;
2017-06-05 23:09:28+0200 [HTTPConnection,39,127.0.0.1] &#39;[!] twisted.internet.error.ConnectionRefusedError Connection was refused by other side&#92;&#92;n&#92;&#92;nConnectionRefusedError: Conne
ction was refused by other side: 111: Connection refused.&#92;&#92;n&#39;
2017-06-05 23:09:28+0200 [HTTPConnection,39,127.0.0.1] &#39;[!] exception mail suppressed for exception (f9527d64c578191cd812f15091ba8f61b2b3cf8321c0ae8f0b08687a861f20a4) [reason: threshold exceede
d]&#39;
2017-06-05 23:09:28+0200 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

evilaliv3 · 2017-06-07 07:41:13 UTC

Thank you @smbd for reporting this.

Do you have by the chance already solved?

We have actually still not reproduced the error so i’m asking you if you could provide some additional information.

The fact that the Onion Service is not working and that in the log you have logs of connections refused make me think that for sure the Tor process is not starting or it is not correctly opening its ports for configuration.

Could you please verify an provide information about the following:

is the Tor process running?
could you please post the output of “/var/globaleaks/torhs# ls -al”
could you please verify if in dmesg you have some error in relation to apparmor?

smbd · 2017-06-07 08:05:52 UTC

Not solved as of now.

Tor is not running according to ps ax | grep tor

total 16
drwx------ 2 debian-tor debian-tor 4096 Jun  4 22:45 .
drwxr-x--- 6 globaleaks debian-tor 4096 Dec 19  2014 ..
-rw------- 1 debian-tor debian-tor   23 Jun  4 22:45 hostname
-rw------- 1 debian-tor debian-tor  887 Dec 19  2014 private_key

dmesg output related to apparmor

[    2.724870] audit: type=1400 audit(1496822002.912:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=638 comm="apparmor_parser"
[    2.724876] audit: type=1400 audit(1496822002.912:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=638 comm="apparmor_parser"
[    2.724881] audit: type=1400 audit(1496822002.912:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=638 comm="apparmor_parser"
[    2.737086] audit: type=1400 audit(1496822002.924:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="system_tor" pid=643 comm="apparmor_parser"
[    2.742934] audit: type=1400 audit(1496822002.928:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="usr.bin.globaleaks" pid=645 comm="apparmor_parser"
[    2.748343] audit: type=1400 audit(1496822002.936:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="usr.bin.tor2web" pid=646 comm="apparmor_parser"
[    2.756924] audit: type=1400 audit(1496822002.944:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/haveged" pid=647 comm="apparmor_parser"
[    2.768579] audit: type=1400 audit(1496822002.956:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=650 comm="apparmor_parser"

smbd · 2017-06-08 14:48:39 UTC

@evilaliv3 any suggestions?

evilaliv3 · 2017-06-08 16:06:13 UTC

still not @smbd

we have never tried to update a platform from a distro version to an other, but always ported the data directory.

try to understand why tor is not started and why it is not able to access the hostname/key files in /var/torhs

the startup of tor is quite completely independent from globaleaks and the only thing that we configure for that is an apparmor script: https://github.com/globaleaks/GlobaLeaks/blob/master/debian/globaleaks.postinst#L45

smbd · 2017-06-08 16:39:57 UTC

@evilaliv3 here are the logs from /var/log/tor/log:

Jun 08 17:28:57.000 [notice] Tor 0.2.7.6 (git-605ae665009853bd) opening log file.
Jun 08 17:28:57.360 [notice] Tor v0.2.7.6 (git-605ae665009853bd) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g and Zlib 1.2.8.
Jun 08 17:28:57.365 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 08 17:28:57.366 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jun 08 17:28:57.366 [notice] Read configuration file "/etc/tor/torrc".
Jun 08 17:28:57.375 [notice] Opening Socks listener on 127.0.0.1:9050
Jun 08 17:28:57.375 [notice] Opening DNS listener on 127.0.0.1:5353
Jun 08 17:28:57.375 [notice] Opening Transparent pf/netfilter listener on 127.0.0.1:9040
Jun 08 17:28:57.375 [notice] Opening Control listener on /var/run/tor/control
Jun 08 17:28:57.000 [warn] Couldn't open "/var/globaleaks/torhs//hostname.tmp" (/var/globaleaks/torhs//hostname) for writing: Read-only file system
Jun 08 17:28:57.000 [warn] Could not write onion address to hostname file.
Jun 08 17:28:57.000 [warn] Error loading rendezvous service keys
Jun 08 17:28:57.000 [err] set_options(): Bug: Acting on config options left us in a broken state. Dying. (on Tor 0.2.7.6 )
Jun 08 17:28:57.000 [notice] Tor 0.2.7.6 (git-605ae665009853bd) opening log file.
Jun 08 17:28:57.972 [notice] Tor v0.2.7.6 (git-605ae665009853bd) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g and Zlib 1.2.8.
Jun 08 17:28:57.973 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 08 17:28:57.973 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jun 08 17:28:57.973 [notice] Read configuration file "/etc/tor/torrc".
Jun 08 17:28:57.981 [notice] Opening Socks listener on 127.0.0.1:9050
Jun 08 17:28:57.981 [notice] Opening DNS listener on 127.0.0.1:5353
Jun 08 17:28:57.982 [notice] Opening Transparent pf/netfilter listener on 127.0.0.1:9040
Jun 08 17:28:57.982 [notice] Opening Control listener on /var/run/tor/control
Jun 08 17:28:57.000 [warn] Couldn't open "/var/globaleaks/torhs//hostname.tmp" (/var/globaleaks/torhs//hostname) for writing: Read-only file system
Jun 08 17:28:57.000 [warn] Could not write onion address to hostname file.
Jun 08 17:28:57.000 [warn] Error loading rendezvous service keys
Jun 08 17:28:57.000 [err] set_options(): Bug: Acting on config options left us in a broken state. Dying. (on Tor 0.2.7.6 )

The problem is with owner and permissions of /var/globaleaks/torhs:

root@x2://var/globaleaks# ls -l /var/globaleaks/
total 16
drwx------ 2 globaleaks globaleaks 4096 Jun  8 18:00 db
drwx------ 7 globaleaks globaleaks 4096 Apr 11  2016 files
drwx------ 2 globaleaks globaleaks 4096 May 16 08:31 log
drwx------ 2 debian-tor debian-tor 4096 Jun  4 22:45 torhs

If I change torhs owner to globaleaks after a restart, it goes back to debian-tor

fpietrosanti · 2017-06-15 12:41:05 UTC

This problem seems to have happened multiple time in past:

Does gl-fix-permissions script fix the problem as a workaround?

evilaliv3 · 2017-06-15 13:32:17 UTC

@fpietrosanti that command is executed at any startup of globaleaks so i do not think that would fix the issue.

i’m so looking forward for the integration of the patch for avoiding to use the filesystem when dealing with Tor.

evilaliv3 · 2017-06-15 13:42:53 UTC

@smbd now tha i rethink to the fact that you told to have updated the distribution, maybe you have resetted the /etc/tor/torrc?

does it still contain this lines?

VirtualAddrNetwork 10.23.47.0/10
AutomapHostsOnResolve 1
TransPort 9040
TransListenAddress 127.0.0.1
DNSPort 5353
DNSListenAddress 127.0.0.1
HiddenServiceDir /var/globaleaks/torhs/
HiddenServicePort 80 127.0.0.1:8082

smbd · 2017-06-15 13:48:33 UTC

@evilaliv3 no, during the upgrade we opted to keep old /etc/tor/torrc. Tor wouldn’t start immediately after the upgrade. Later we deleted the old /etc/tor/torrc and pasted only GL config below into the new file. It still did not work.

# BEGIN GlobaLeaks Configuration - DO NOT EDIT!
VirtualAddrNetwork 10.23.47.0/10
AutomapHostsOnResolve 1
TransPort 9040
TransListenAddress 127.0.0.1
DNSPort 5353
DNSListenAddress 127.0.0.1
HiddenServiceDir /var/globaleaks/torhs/
HiddenServicePort 80 127.0.0.1:8082
# END GlobaLeaks Configuration - DO NOT EDIT!

nskelsey · 2017-06-15 13:54:49 UTC

Hey @smbd

I think you can just copy /var/globaleaks/torhs directory (with perms for debian-tor) outside of /var/globaleaks and change the following line in /etc/tor/torrc to the new path:

HiddenServiceDir /path/to/torhs

This is a proposed work around. . . but because GL only needs the config once (on system init) it seems ridiculous to me that it is blocking normal startup+restarts.

nskelsey · 2017-06-20 16:04:28 UTC

Hi @smbd we believe we have identified the issue on 16.04 and have a fix.

systemd has added its own sandboxing that silently controls some permissions on the system. You can run the following commands which come from globaleaks.posinst and it should resolve the issue.

mkdir -p /etc/systemd/system/tor\@default.service.d
# Add Systemd rules for allowing Tor to access to directory /var/globaleaks/torhs/
cat <<EOF >> /etc/systemd/system/tor\@default.service.d/directory.conf
[Service]
ReadWriteDirectories=-/var/globaleaks/torhs/
EOF

systemctl daemon-reload

I ran into this on a similar 16.04 instance today and it resolved the issue. Let us know if it works for you.

smbd · 2017-06-20 21:51:33 UTC

@nskelsey Yes, this fixed it. Thanks a lot.

Now another issue. Our Letsencrypt certs expired. We renewed them, changed and verified all symlinks.

However this is what we see in the logs now:

2017-06-20 23:48:31+0200 [-] twistd 16.0.0 (&#47;usr&#47;bin&#47;python 2.7.12) starting up.
2017-06-20 23:48:31+0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2017-06-20 23:48:31+0200 [-] Site starting on 80
2017-06-20 23:48:31+0200 [-] Starting factory &lt;twisted.web.server.Site instance at 0x7fa6fc700e18&gt;
2017-06-20 23:48:31+0200 [-] Site starting on 8082
2017-06-20 23:48:31+0200 [-] Unhandled error in Deferred:
2017-06-20 23:48:31+0200 [-] [E] Unhandled exception raised:
2017-06-20 23:48:31+0200 [-] [E] globaleaks.utils.tls.ValidationException None&#92;n&#92;nValidationException: The certficate has expired&#92;n
2017-06-20 23:48:31+0200 [-] Starting factory &lt;txsocksx.client.SOCKS5ClientFactory instance at 0x7fa6fd030b48&gt;
2017-06-20 23:48:31+0200 [-] Starting factory &lt;txsocksx.client.SOCKS5ClientFactory instance at 0x7fa6fd030f80&gt;
2017-06-20 23:48:31+0200 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: globaleaks.utils.tls.ValidationException: The certficate has expired

2017-06-20 23:48:31+0200 [-] GlobaLeaks is now running and accessible at the following urls:
2017-06-20 23:48:31+0200 [-] - http:&#47;&#47;0.0.0.0:8082
2017-06-20 23:48:31+0200 [-] - http:&#47;&#47;secure.***********.org:8082
2017-06-20 23:48:31+0200 [-] - http:&#47;&#47;***************.onion
2017-06-20 23:48:32+0200 [-] Starting factory &lt;txsocksx.client.SOCKS5ClientFactory instance at 0x7fa6fd16f1b8&gt;
2017-06-20 23:48:33+0200 [-] Stopping factory &lt;txsocksx.client.SOCKS5ClientFactory instance at 0x7fa6fd16f1b8&gt;
2017-06-20 23:48:35+0200 [-] Stopping factory &lt;txsocksx.client.SOCKS5ClientFactory instance at 0x7fa6fd030f80&gt;
2017-06-20 23:48:35+0200 [-] Stopping factory &lt;txsocksx.client.SOCKS5ClientFactory instance at 0x7fa6fd030b48&gt;

nskelsey · 2017-06-21 10:31:55 UTC

@smbd the certificate and keys for SSL are loaded in the database when using the HTTPS built into GL. You need to access the platform from the administration interface using the Tor browser, disable HTTPS, replace the expired lets encrypt certificate with the new one and re-enable HTTPS.

Right now, we are finished up a feature to automate the renewal using Let’s Encrypt. It will go live in v2.68.0 before the end of June.

smbd · 2017-06-21 11:08:30 UTC

@smbd although both tor and globaleaks are up now (confirmed by ps) and firewall is disabled sudo ufw disable, our onion site is still inaccessible via Tor browser

nskelsey · 2017-06-21 13:26:20 UTC

This is unfortunate, sorry it is such a challenge to get the service up. I have 3 questions:

Is GL responding on its local port?

curl localhost:8082

Is tor listening on 9050?

netstat -tulpen

Is the tor hidden service configured with the expected onion address?

# check tor log and syslog
less /var/log/tor/log /var/log/syslog
# check config
less /etc/tor/torrc
# check onion address
cat /var/globaleaks/torhs/hostname

smbd · 2017-06-21 13:53:43 UTC

yes

x2:~# curl localhost:8082
x2:~#

yes

x2:~# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
...
tcp        0      0 127.0.0.1:9050          0.0.0.0:*               LISTEN      0          14479       1053/tor
....

x2:~# cat /var/log/tor/log
......
Jun 21 13:09:25.000 [notice] Interrupt: exiting cleanly.
Jun 21 13:09:35.000 [notice] Tor 0.2.7.6 (git-605ae665009853bd) opening log file.
Jun 21 13:09:35.636 [notice] Tor v0.2.7.6 (git-605ae665009853bd) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g and Zlib 1.2.8.
Jun 21 13:09:35.642 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 21 13:09:35.643 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jun 21 13:09:35.643 [notice] Read configuration file "/etc/tor/torrc".
Jun 21 13:09:35.654 [notice] Opening Socks listener on 127.0.0.1:9050
Jun 21 13:09:35.655 [notice] Opening DNS listener on 127.0.0.1:5353
Jun 21 13:09:35.656 [notice] Opening Transparent pf/netfilter listener on 127.0.0.1:9040
Jun 21 13:09:35.656 [notice] Opening Control listener on /var/run/tor/control
Jun 21 13:09:35.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
Jun 21 13:09:35.000 [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
Jun 21 13:09:36.000 [notice] Bootstrapped 0%: Starting
Jun 21 13:09:36.000 [notice] Bootstrapped 5%: Connecting to directory server
Jun 21 13:09:36.000 [notice] Bootstrapped 80%: Connecting to the Tor network
Jun 21 13:09:36.000 [notice] Signaled readiness to systemd
Jun 21 13:09:36.000 [notice] Bootstrapped 85%: Finishing handshake with first hop
Jun 21 13:09:37.000 [notice] Bootstrapped 90%: Establishing a Tor circuit
Jun 21 13:09:38.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working.
Jun 21 13:09:38.000 [notice] Bootstrapped 100%: Done

x2:~# cat /var/log/syslog
......
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.636 [notice] Tor v0.2.7.6 (git-605ae665009853bd) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g and Zlib 1.2.8.
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.642 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.643 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.643 [notice] Read configuration file "/etc/tor/torrc".
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.654 [notice] Opening Socks listener on 127.0.0.1:9050
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.655 [notice] Opening DNS listener on 127.0.0.1:5353
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.656 [notice] Opening Transparent pf/netfilter listener on 127.0.0.1:9040
Jun 21 13:09:35 x2 tor[1053]: Jun 21 13:09:35.656 [notice] Opening Control listener on /var/run/tor/control
Jun 21 13:09:36 x2 systemd[1]: Started Anonymizing overlay network for TCP.
Jun 21 13:09:38 x2 globaleaks[1011]:  * Starting GlobaLeaks daemon globaleaks
Jun 21 13:09:38 x2 globaleaks[1011]:  * Enabling GlobaLeaks Apparmor Sandboxing...
Jun 21 13:09:40 x2 globaleaks[1011]:    ...done.
Jun 21 13:09:40 x2 systemd[1]: Started LSB: Start the GlobaLeaks server..
Jun 21 13:09:40 x2 systemd[1]: Reached target Multi-User System.
Jun 21 13:09:40 x2 systemd[1]: Reached target Graphical Interface.
Jun 21 13:09:40 x2 systemd[1]: Starting Update UTMP about System Runlevel Changes...
Jun 21 13:09:40 x2 systemd[1]: Started Update UTMP about System Runlevel Changes.
Jun 21 13:09:40 x2 systemd[1]: Startup finished in 1.643s (kernel) + 7.573s (userspace) = 9.217s.

x2:~# cat /etc/tor/torrc
# BEGIN GlobaLeaks Configuration - DO NOT EDIT!
VirtualAddrNetwork 10.23.47.0/10
AutomapHostsOnResolve 1
TransPort 9040
TransListenAddress 127.0.0.1
DNSPort 5353
DNSListenAddress 127.0.0.1
HiddenServiceDir /var/globaleaks/torhs
#HiddenServiceDir /var/torhs
HiddenServicePort 80 127.0.0.1:8082
# END GlobaLeaks Configuration - DO NOT EDIT!

x2:~# cat /var/globaleaks/torhs/hostname
rfftlkqzjdse5jvl.onion

nskelsey · 2017-06-21 15:04:42 UTC

Okay, its a bug within GL that got shipped in v2.67.5 we just released a fix in v.2.67.9. Update the globaleaks package and you should be able to connect.

There was an erroneous redirect preventing the interface from loading.

smbd · 2017-06-21 16:36:11 UTC

onion is finally back up, but not https. Need to re-upload new certs probably.

smbd · 2017-06-23 10:57:27 UTC

@nskelsey Now we cannot update SSL because we cannot log in as admin to Onion.

The message says Error! The resource is available only via Tor browser.

The problem is that WE ARE using Tor browser when trying to log in.