I believe so, but I was able to get around it and complete my latest import with the workaround I mentioned above.
I'll close this issue for now then. If anyone else stumbles upon it they can reopen or just create a new one 👍
~/shared/device_sync$ ./nb-dt-import.py
Package devicetype-library is already installed, updating /srv/netbox/shared/device_sync/repo
104 Vendors Found
2306 Device-Types Found
Traceback (most recent call last):
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connection.py", line 454, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connection.py", line 454, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/netbox/shared/device_sync/./nb-dt-import.py", line 53, in <module>
main()
File "/srv/netbox/shared/device_sync/./nb-dt-import.py", line 25, in main
netbox.create_device_types(device_types)
File "/srv/netbox/shared/device_sync/netbox_api.py", line 97, in create_device_types
self.device_types.create_interfaces(device_type["interfaces"], dt.id)
File "/srv/netbox/shared/device_sync/netbox_api.py", line 196, in create_interfaces
existing_interfaces = {str(item): item for item in self.netbox.dcim.interface_templates.filter(
File "/srv/netbox/shared/device_sync/netbox_api.py", line 196, in <dictcomp>
existing_interfaces = {str(item): item for item in self.netbox.dcim.interface_templates.filter(
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/pynetbox/core/response.py", line 127, in __next__
next(self.response), self.endpoint.api, self.endpoint
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/pynetbox/core/query.py", line 280, in get
req = self._make_call(add_params=add_params)
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/pynetbox/core/query.py", line 230, in _make_call
req = getattr(self.http_session, verb)(
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
The request logs show the following when it works:
[pid: 2757|app: 0|req: 109/109] 127.0.0.1 () {30 vars in 467 bytes} [Fri Jun 2 17:45:43 2023] GET /api/dcim/manufacturers/?limit=0 => generated 360 bytes in 84 msecs (HTTP/1.1 200) 9 headers in 340 bytes (1 switches on core 0)
And the following when it doesn't:
[pid: 2757|app: 0|req: 104/104] 127.0.0.1 () {30 vars in 365 bytes} [Fri Jun 2 17:44:44 2023] GET /api/ => generated 455 bytes in 33 msecs (HTTP/1.1 200) 10 headers in 332 bytes (1 switches on core 0)
If I manually replicate using pynetbox, the API calls work.
$ python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynetbox
>>> nb = pynetbox.api("http://localhost", "xxxx")
>>> {str(item): item for item in nb.dcim.manufacturers.all()}
{'blah': blah}Trying to use the NetBox class gets me the above:
>>> from netbox_api import NetBox
>>> import settings
Package devicetype-library is already installed, updating /srv/netbox/shared/device_sync/repo
>>> nb2 = NetBox(settings)
Traceback (most recent call last):
Replicating just the get_manufacturers function gets me the actual data:
>>> def get_manufacturers():
... return {str(item): item for item in nb.dcim.manufacturers.all()}
>>> get_manufacturers()
{'blah': blah}I can open a new issue if not related.
I'm getting the exact same error on my test netbox instance.
Netbox is running as a docker instance (using lscr.io/linuxserver/netbox:latest) on Ubuntu 20.04 host. The device type library import scripts running natively on the host in a venv.
It imports some devices then errors out. Watching the logs from the netbox container, I can see the API request counter increasing - the most recent time it managed 350 before crashing out, the time before that only 11 requests.
I am receiving this error as well. I am using the docker container to perform the import. I have modified what @ndom91 provided in order to get items to import. Sometimes it fails quickly, other times it manages to import a large number of records:
Workaround Script:
Edit: I ended up having to loop over each vendor one at a time; even then there were constant errors. However, after enough retries, I managed to get all the device types imported.
#!/bin/bash
serverOrIp=$1
port=$2
token=$3
vendorList="3com,a10,adva,apc,avm,actiontec,adtran,albis-elcon,alcatel-lucent,\
allied telesis,alpha,apple,arista,avocent,brocade,calix,canon,certa,check point,\
ciena,cisco,citrix,colable,corning,cryptopro,cyberpower,d-link,datadirect networks,\
datacom,datto,deciso,dell,delta,digi,digital loggers,eaton,edgecore,eltex,engenius,\
ericsson,extreme networks,f5,fs,factor-ts,firebrick,fortinet,fujitsu,generic,hpe,huawei,\
ibm,infinera,infoblox,infotecs,inspur,intel,juniper,lancom,lenovo,leviton,mellanox,\
mikrotik,mimosa networks,nec,neousys,netapp,netgate,netgear,netonix,nokia,nvidia,opengear,\
pc engines,palo alto,panduit,parks,peplink,pluribus,qct,qnap,qtech,qotom,rad,raritan,\
raspberry pi,riello,riverbed,rockwell automation,rohde & schwarz,ruckus,snr,solid optics,\
sonicwall,sophos,supermicro,synology,tp-link,teltonika,testing,thales,trendnet,ubiquiti,\
v-solution,vertiv,wti,yamaha,zpe,zte,zyxel,ghipsystem"
IFS=',' read -ra vendorsArray <<< "$vendorList"
echo "Starting outer loop.."
for vendor in "${vendorsArray[@]}"; do
echo "Starting inner loop for vendor: $vendor"
echo "docker run --rm -e \"NETBOX_URL=http://${serverOrIp}:${port}/\" -e \"NETBOX_TOKEN=${token}\" -e \"VENDORS=${vendor}\" ghcr.io/minitriga/netbox-device-type-library-import"
until docker run --rm -e "NETBOX_URL=http://${serverOrIp}:${port}/" -e "NETBOX_TOKEN=${token}" -e "VENDORS=${vendor}" ghcr.io/minitriga/netbox-device-type-library-import; [[ $? -eq 0 ]];
echo "*** RETRYING IMPORT ***"
sleep 3s
echo "Netbox Device-Type Import successful for vendor: $vendor"
echo "All vendors processed"
Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
response.begin()
File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
response.begin()
File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/nb-dt-import.py", line 853, in <module>
main()
File "/app/nb-dt-import.py", line 826, in main
createDeviceTypes(deviceTypes, nb)
File "/app/nb-dt-import.py", line 694, in createDeviceTypes
createConsoleServerPorts(
File "/app/nb-dt-import.py", line 348, in createConsoleServerPorts
all_consoleserverports = {str(item): item for item in nb.dcim.console_server_port_templates.filter(devicetype_id=deviceType)}
File "/usr/local/lib/python3.9/site-packages/pynetbox/core/endpoint.py", line 225, in filter
ret = [self._response_loader(i) for i in req.get()]
File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 300, in get
return req_all()
File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 279, in req_all
req = self._make_call(add_params=add_params)
File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 245, in _make_call
req = getattr(self.http_session, verb)(
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I wonder if it is the docker bit that is messing with it. We are still working on updating those docker images since they havent been rebuilt since the last maintainer
No I am facing the same issue while using standalone python script on CentOS 7 VM
Silly workaround - a bash script to retry importing until the exit code is 0, i.e. success 😂
#!/bin/bash
echo "Entering python venv"
source "$(pwd)/bin/activate"
echo "Starting loop.."
until python3 nb-dt-import.py; [[ $? -eq 0 ]];
echo "*** RETRYING IMPORT ***"
sleep 1
echo "Netbox Device-Type Import successful"
This works until there is some big number added, after that the script fails while going through existing deviecs, i.e. not reaching those that are not added. A workaround is to randomize the list of device types that need to be added :). Altogether this is a silly solution, but it works:)
So, int the netbox_api.py file add import random and then randomize 'device_types_to_add' within the create_device_types method, like this:
import random
def create_device_types(self, device_types_to_add):
random.shuffle(device_types_to_add) # <--- Just add this line
for device_type in device_types_to_add:
.................
